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Summary 


College education is fundamental to students’ upward mobility, states’ economic growth, 
and the country’s economic competitiveness (Bureau of Labor Statistics, 2015; Hanushek & 
Kimko, 2000; Hanushek & Woessmann, 2012). Researchers have forecast that 63 percent 
of future jobs will require a college degree, yet in the coming years the United States will 
likely produce 3 million fewer college graduates than are needed to fill workforce demand 
(Carnevale, Smith, & Stohl, 2010). With these statistics and projections in mind, policy' 
makers are placing greater emphasis on motivating high school students to attend college 
and on ensuring that students have the skills needed to succeed in college. Minnesota leg- 
islators, for example, have called on the public K-12 education system to motivate middle 
school students to attend college and to help high school students plan for college as a 
means of attaining their career aspirations (Grow, 2013). 

To better enable middle and high schools to increase college participation and success rates 
among their students, the University of Minnesota’s College Readiness Consortium deveh 
oped Ramp-Up to Readiness™ (Ramp-Up), a schoolwide advisory program to increase 
students’ likelihood of college enrollment and completion by enhancing five dimensions 
of college readiness (academic, admissions, career, financial, and personal-social) among 
middle school and high school students. As of 2016, the program has been rolled out in 
150 middle and high schools throughout Minnesota, but little information is available on 
the program’s effectiveness. 

Members of the Midwest College and Career Success Research Alliance expressed an 
interest in learning how the program attempts to improve students’ college readiness, how 
it differs from typical college-readiness supports in high schools, how it is implemented, 
whether schools meet the consortium’s expectations for implementation, how school staff 
perceive the program, and whether the program has an immediate impact on student out- 
comes. Since 2012 Regional Educational Laboratory Midwest has worked with alliance 
members to answer these questions. 

This report describes a study of the impacts of the Ramp-Up program after one year of 
implementation and provides information on how Ramp-Up differs from college-related 
supports in other schools and the degree to which Ramp-Up has been implemented with 
fidelity. 

This study involved 49 public high schools serving grades 10-12 in Minnesota and western 
Wisconsin. Twenty-five of the 49 schools were randomly assigned to implement Ramp-Up 
during the 2014/15 academic year (Ramp-Up schools), and the other 24 schools were ran- 
domly assigned to continue with business as usual during 2014/15 and then implement 
Ramp-Up during 2015/16 (comparison schools). The final analytic sample was smaller than 
expected, which made it harder to detect program impacts. Data collected in fall 2014 and 
spring 2015 were used to examine the impact of the program after one year of implementa- 
tion on students’ scores on the ACT Engage goal striving and commitment to college scales 
along with their likelihood of completing the Free Application for Federal Student Aid 
[FAFSA] and submitting at least one college application. The study also addressed ques- 
tions about whether the types of college-readiness supports offered by Ramp-Up schools 
differed from those offered by comparison schools, whether staff in Ramp-Up schools 


engaged in more college-oriented activity, and whether Ramp-Up schools in 2014/15 were 
able to implement the program adequately by consortium standards. 

The study’s main findings are: 

• After a single year of implementation, there were no statistically significant dif- 
ferences on self-reported goal striving or commitment to college scores or on like- 
lihood of completing the FAFSA and submitting at least one college application 
between students in Ramp-Up schools and students in comparison schools. 

• Ramp-Up schools and comparison schools offered the same types of supplemental 
college-readiness supports. 

• Staff in Ramp-Up schools engaged in more college-readiness activity than did staff 
in comparison schools. Students in Ramp-Up schools perceived a greater emphasis 
among staff on two of the five dimensions of college readiness (admissions readi- 
ness and financial readiness) than did students in comparison schools. 

• When averaged across program components, 96 percent of Ramp-Up schools’ 
implementation scores fell within the range that the program developer clas- 
sified in advance as adequate. However only 3 of the 25 (12 percent) Ramp-Up 
schools had adequate scores for all five of Ramp-Up’s key components (structural 
supports, professional development, curriculum delivery, curriculum content, and 
postsecondary planning tools), suggesting that Ramp-Up schools need to improve 
implementation if they hope to produce the program’s intended impacts. 


Contents 


Acknowledgments i 

Summary ii 

Why this study? 1 

What the study examined 4 

Overview of Regional Educational Laboratory Midwest’s two studies of Ramp-Up 4 

Research questions for the current study 4 

What the study found 8 

After the first year of implementation, students in Ramp-Up schools and students in 
comparison schools were equally likely to complete the Free Application for Federal 
Student Aid and had similar levels of personal college readiness 8 

After the first year of implementation, students in Ramp-Up schools and students in 

comparison schools were equally likely to submit at least one college application 8 

After the first year of implementation, Ramp-Up had no impacts on immediate and 

exploratory outcomes among subgroups of students 9 

Ramp-Up schools and comparison schools offered the same supplemental college-readiness 
supports, but staff in Ramp-Up schools showed more college-focused activity than did 
staff in comparison schools 10 

Despite school staff members’ enhanced engagement in college-oriented activities in 
Ramp-Up schools, implementation of Ramp-Up components after the first year was 
inadequate for many schools 11 

Implications of the study findings 14 

Limitations of the study 16 

Size and integrity of the analytic samples 16 

The outcome measures 17 

Inferences that can be drawn after a single year of program implementation 17 

Generalizability of the findings 17 

Appendix A. Ramp-up program history A-l 

Appendix B. Methods B-l 

Appendix C. Detailed results C-l 

Notes Notes-1 

References Ref-1 

Boxes 

1 Ramp-Up to Readiness™ program description 2 

2 Data and methods 6 


IV 


Figures 

1 Ramp-Up to Readiness theory of action 2 

2 Students in Ramp-Up schools and students in comparison schools showed little 

difference in immediate outcomes, 2014/15 9 

3 According to staff responses, the percentage of students who received assistance with 
key enrollment actions was higher in Ramp-Up schools than in comparison schools, fall 

2014 12 

4 Staff in Ramp-Up schools were more likely than staff in comparison schools to engage 

in college-oriented activities, 2014/15 12 

5 Students in Ramp-Up schools observed significantly more staff support for two of the 

five dimensions of college readiness than did students in comparison schools, 2014/15 13 

6 For Ramp-Up schools, overall implementation was excellent or adequate, but 

implementation for some schools on some components was inadequate, 2014/15 13 

B1 Process used to randomly assign schools to conditions B-2 

B2 According to the developer of Ramp-Up, for the program to have effects, schools need 

to implement the program’s key components and subcomponents B-15 

Cl Change in predicted probability of completing the Free Application for Federal Student 
Aid and submitting at least one college application between students in Ramp-Up 
schools and students in comparison schools, 2014/15 C-7 

Tables 

1 Staff in Ramp-Up schools and comparison schools reported offering the same basic 

college -readiness supports, fall 2014 (percent of staff member respondents) 11 

B1 Number of students in grades 10-12 and total sample size by treatment condition, 

fall 2014 B-2 

B2 Data used to address each research question B-3 

B3 Student survey and ACT Engage survey response rates for fall 2014 and spring 2015, by 

treatment condition B-5 

B4 Staff survey response rates, by treatment condition, 2014/15 B-7 

B5 Rates of missing data in the analytic sample for extant variables by treatment condition, 

2014/15 B-8 

B6 Attrition rates for full samples and subsamples on all outcomes B-9 

B7 Power calculations B-10 

B8 Description of impact analysis models, by research question B-12 

Cl At baseline, Ramp-Up schools and comparison schools had student populations with 

similar demographic characteristics and student achievement, 2013/14 C-l 

C2 Regression models predicting self-reported completion of the Free Application for 
Federal Student Aid, ACT Engage goal striving and commitment to college scale 
scores, and submission of at least one college application among all eligible students, by 
student and school characteristics C-2 

C3 Regression models predicting self-reported completion of the Free Application for 
Federal Student Aid, ACT Engage goal striving and commitment to college scale 
scores, and submission of at least one college application among students between the 
33rd and 66th percentiles of math achievement C-3 

C4 Regression models predicting self-reported completion of the Free Application for 
Federal Student Aid, ACT Engage goal striving and commitment to college scale 
scores, and submission of at least one college application among students at or above 
the 66th percentile of math achievement C-4 


v 


C5 Regression models predicting self-reported completion of the Free Application for 

Federal Student Aid, ACT Engage goal striving and commitment to college scores, and 
submission of at least one college application among students eligible for the federal 
school lunch program C-5 

C6 Ramp-Up schools’ implementation scores show adequate fidelity overall, but some 

components and subcomponents were inadequately implemented, 2014/15 C-6 


VI 


Why this study? 


College education is fundamental to students’ upward mobility, states’ economic growth, 
and the country’s economic competitiveness (Bureau of Labor Statistics, 2015; Hanushek 
& Kimko, 2000; Hanushek & Woessmann, 2012). Ninety-three percent of graduating 
seniors plan to enroll in college (Ross et al, 2012), yet 79 percent of students in the United 
States do so by age 20 (Center for Public Education, 2014). Of the students who enroll, 
only 65 percent attain a postsecondary certificate or degree (Snyder & Dillow, 2015). 

What obstacles do high school students face that prevent them from actualizing their 
college aspirations? Some high school seniors who plan to go to college struggle with com- 
pleting financial aid applications (such as the Free Application for Federal Student Aid 
[FAFSA]) or submitting college applications (Avery & Kane, 2004; Roderick, Nagaoka, 
Coca, & Moeller, 2008). Other students may be among the 30 percent of applicants who 
fail to gain admittance to their desired college because of poor scores on college admis- 
sions tests or low grade point average or class rank (Clinedinst, 2015). 

Faced with the gap between high school students’ college aspirations and the actual per- 
centage of students who attain a postsecondary certificate or degree, policymakers are 
expecting K-12 school systems to better prepare students to enroll and succeed in college. 

To help high schools in this endeavor, the What Works Clearinghouse convened a panel 
of experts to offer five recommendations for helping students obtain postsecondary educa- 
tion outcomes. 1 The panel recommended that schools offer students more rigorous courses 
and curricula, use assessments to track students’ progress on admissions criteria, group stu- 
dents with others who aspire for a college education, assist students with the college admis- 
sions process, and increase families’ understanding of the financial burden associated with 
college and of how to apply for financial aid (Tierney, Bailey, Constantine, Finkelstein, & 
Hurd, 2009). Empirical support for these recommendations is lacking, however. The What 
Works Clearinghouse found that only 16 of 500 studies related to these recommended 
practices met its standards for evidence (Tierney et al., 2009). 

For many policymakers and practitioners, more evidence on the benefits of these practices 
is needed. One group of stakeholders that seeks to better understand the processes and 
interventions that enhance college going, college completion, and workforce success is the 
Midwest College and Career Success Research Alliance. 2 Alliance members are especially 
interested in learning more about a college-readiness program that incorporates most of 
the What Works Clearinghouse’s recommended practices: the University of Minnesota’s 
College Readiness Consortium Ramp-Up to Readiness™ program (Ramp-Up). The con- 
sortium developed Ramp-Up as a schoolwide guidance program to increase middle and 
high school students’ likelihood of college enrollment and completion by enhancing five 
dimensions of college readiness: academic, admissions, career, financial, and personal- 
social (box 1; see appendix A for a more detailed description of the program and a timeline 
of its implementation in Minnesota and Wisconsin high schools). 

Because each of Ramp-Up’s five dimensions of college readiness is aligned with practic- 
es recommended by the What Works Clearinghouse and supported by a limited number 
of studies (mostly correlational; see appendix A of Lindsay, Davis, Stephan, Bonsu, & 
Narlock, 2016, for a review of the literature), alliance members think that the program 
shows promise for increasing students’ college readiness and their likelihood of college 
enrollment and completion (see figure 1 for Ramp-Up’s theory of action). However, the 
program as a whole has yet to be evaluated. 
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Figure 1. Ramp-Up to Readiness theory of action 
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Ramp-Up 
impact study 


Note: With high fidelity implementation and improvements in the dimensions of college readiness, the consor- 
tium expects immediate outcomes after one year of exposure to Ramp-Up, intermediate outcomes after two 
years of exposure to Ramp-Up, and long-term outcomes after three years exposure to Ramp-Up. 

a. The University of Minnesota's College Readiness Consortium. 

Source: The University of Minnesota's College Readiness Consortium. 


Box 1. Ramp-Up to Readiness™ program description 

Ramp-Up to Readiness™ is a research-based school guidance intervention developed by the University of Minneso- 
ta’s College Readiness Consortium. It consists of a guidance curriculum for middle school and high school students, 
a set of tools to help students set college goals and track progress, and professional development for implementa- 
tion teams and teachers. The theory of action that underlies the program (see figure 1 in the main text) posits that 
increasing students’ knowledge and skills along five dimensions of college readiness (academic, admissions, career, 
financial, and personal-social; defined below) will increase their likelihood enrolling and succeeding in college. 

Core components of Ramp-Up 

Ramp-Up involves five core components, each of which includes a set of resources and activities provided by the 
consortium and enacted in schools: 

• Structural supports. Schools need to establish a Ramp-Up leadership team; appoint a Ramp-Up coordinator; 
enlist homeroom teachers to become Ramp-Up advisors; provide students with opportunities to enroll in 
advanced courses; provide support for professional development for administrators, teachers, and counselors; 
offer time for preparing and conducting advisory sessions and workshops; and adopt a technology platform 
for creating, storing, and sharing students’ postsecondary plans. Ramp-Up leadership teams consist of the 
principal, one counselor, one teacher, and any other suitable individuals. Their responsibilities include creating 
an annual plan and implementation calendar, guiding and monitoring implementation, attending training and 

(continued) 


2 












Box 1. Ramp-Up to Readiness™ program description (continued) 


workshops offered by the program developer, and providing professional development to all staff who deliver the 
program. 

• Professional development. Schools’ Ramp-Up leadership teams and coordinators need to participate in off-site 
professional development sessions led by the consortium, and school Ramp-Up leadership teams and coordina- 
tors must provide on-site professional development to staff who serve as Ramp-Up advisors. 

• Curriculum delivery. Ramp-Up advisors must have the necessary materials and information to understand the 
Ramp-Up curriculum, and they need to facilitate the 28 advisory sessions and 5 workshops at the high school 
with the students in their homerooms. 

• Curriculum content: The content of the Ramp-Up advisory sessions and workshops must cover all five dimensions 
of college readiness. If Ramp-Up advisors follow the activities and workshops provided in their grade-specific 
advisor guides, students should perceive staff-student interactions related to each of the five dimensions. 

• Postsecondary planning tools: Ramp-Up advisors and students need to create a postsecondary plan, complete 
the consortium’s college-readiness rubric, review and update the plan and rubric given students’ experiences, 
and communicate students’ progress to their parents. 

Dimensions of college readiness 

The program developer hypothesizes that Ramp-Up’s curriculum, tools, and professional development will increase 
college readiness by teaching skills and providing information, assistance, and encouragement in five interrelated 
dimensions of college readiness (College Readiness Consortium, 2012): 

• Academic readiness: “The student has the knowledge and skills to do first-year, credit-bearing, college-level 
work” (p. 9). 

• Admissions readiness: “The student has completed all requirements for admission to the type of postsecondary 
education that is a match for their goals, interests and abilities” (p. 9). 

• Career readiness: “The student understands how education increasingly determines income and opportunity in 
the global knowledge economy, and will know which types of jobs in the future will need skilled workers, will pay 
enough to support a family and might be a good match for their interests and abilities” (p. 10). 

• Financial readiness: “Students will be able to cover the cost for one term of study [that is, a degree program] at 
a postsecondary institution through savings, loans, work-study, and financial aid" (p. 10). 

• Personal and social readiness: “The student knows how to set educational goals, make progress toward those 
goals, and create relationships with peers and adults that support the achievement of those goals” (p. 11). 

Outcomes 

According to Ramp-Up’s theory of action, the program has immediate, intermediate, and long-term outcomes. Imme- 
diate outcomes are those that the developer believes can be achieved after a single school year of exposure to 
the program and include completing the Free Application for Federal Student Aid (for high school seniors), taking 
college entrance exams, and developing and pursuing personal goals and commitment to go to college. Intermediate 
outcomes are those that the developer believes can be achieved after two years of exposure to the program and 
include improved grades and state standardized test scores. Long-term outcomes are those that the developer 
believes can be achieved after three years of exposure to the program and include increased likelihood of enrolling 
in a two- or four-year college, decreased likelihood of needing remedial coursework in college, and increased likeli- 
hood of persisting in college. 

Impacts of the program result, theoretically, from the cumulative growth across all five dimensions of readiness. 
No one-to-one correspondence is hypothesized between the outcomes and the college-readiness dimensions. 
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What the study examined 


Members of the alliance partnered with REL Midwest to answer the questions most fre- 
quently asked about Ramp-Up by school and school district administrators, which include: 

• What is the program, and how is it different from what is currently offered? 

• What does implementation entail for schools, and have schools been able to 
implement the program to the developer’s specifications? 

• What do educators consider Ramp-Up’s strengths and weaknesses? 

• Does the program have an impact on student outcomes? 

Overview of Regional Educational Laboratory Midwest’s two studies of Ramp-Up 

A previous REL Midwest study described Ramp Up and how it supplements other col- 
lege-readiness supports, documented the degree to which schools were able to implement 
the program during a single year, and summarized perceptions of the program among 
school staff (Lindsay et al., 2016). That study, referred to here as the Ramp-Up implemen- 
tation study, included 20 schools in Minnesota. Ten schools in that study implemented 
Ramp-Up immediately (2013/14), and 10 schools delayed implementation by one year. The 
Ramp-Up implementation study addressed the first three questions above (through inter- 
views and focus groups) but was unable to look at program impact. 

The current study, referred to as the impact study, examined the immediate impacts of the 
program on students’ college enrollment actions and self-reported personal readiness (figure 
1) by randomly assigning an additional 50 schools (beyond the original 20 schools from 
the Ramp-Up implementation study) to either implement Ramp-Up immediately (during 
the 2014/15 school year) or to delay implementation until the following year (2015/16). 3 
The current study examines implementation only to give context to the impact estimates. 

The Ramp-Up implementation study found that the program supplements the college-re- 
lated supports that schools already tend to have in place (for example, access to dual-credit 
courses and Upward Bound, college visits, financial aid nights) while engaging with stu- 
dents at earlier grade levels to begin developing the five dimensions of readiness and to 
enhance students’ likelihood of participating in other college-related supports (Lindsay, 
et. al, 2016). The implementation study also showed that schools were generally able to 
implement Ramp-Up (average implementation ratings showed “adequate” fidelity) but 
that Ramp-Up advisors in 80 percent of schools did not adequately implement the post- 
secondary planning tools component of Ramp-Up (assisting students in creating their 
postsecondary plans, tracking students’ progress on the college-readiness rubric, and com- 
municating students’ progress to their parents; Lindsay et al., 2016). 

Research questions for the current study 

This impact study focuses on whether schools and districts that adopt Ramp-Up are likely 
to see an impact on student outcomes after the first year of implementation. 

This study addresses two confirmatory research questions: 

• What is the impact of Ramp-Up on the likelihood of grade 12 students complet- 
ing the LALSA? 

• What is the impact of Ramp-Up on grade 10, 11, and 12 students’ scores on the 
ACT Engage goal striving and commitment to college scales? 
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The study also addressed two exploratory research questions: 

• What is the impact of Ramp-Up on the likelihood of grade 12 students submitting 
at least one college application? 

• What is the impact of Ramp-Up on ACT Engage goal striving and college com- 
mitment scale scores, the likelihood of FAFSA completion, and the likelihood of 
submitting at least one college application for students who scored in the upper 
third on the grade 8 state standardized math test, students who scored in the 
middle third on the grade 8 state standardized math test, and students who are 
eligible for the federal school lunch program? 

These questions were exploratory because the program developer had no firm expectations 
that these outcomes would be affected by the program within a single year. 

The final research question examined variation in the impact of Ramp-Up on the four out- 
comes for students who scored in the upper third and those who scored in the middle third 
on the grade 8 state standardized math test. Students in the upper third were likely to have 
completed key college actions regardless of participation in Ramp-Up, possibly diluting 
program impacts, so the study team looked at the upper and middle tertiles separately to 
determine whether there were differential impacts by prior level of achievement. In addi- 
tion, the study examined differences in the impact of Ramp-Up on the four outcomes for 
students who are eligible for the federal school lunch program (a proxy for economic disad- 
vantage). Prior research suggests that low-income students or those whose parents did not 
attend college have greater needs for assistance in the college-enrollment process and less 
access to academic opportunities to prepare them for college (Avery & Kane, 2004; Fareau 
& Wieninger, 2008; McDonough, 1997). Because Ramp-Up attempts to target those needs, 
the impact of Ramp-Up may be strongest among students who would be less likely to con- 
sider attending college because of their family’s financial status. An analysis of Ramp-Up’s 
impact on students eligible for the federal school lunch program could provide evidence of 
whether Ramp-Up had an impact on this subgroup. 

To establish context surrounding the estimates of Ramp-Up’s immediate impacts, the study 
also examined three implementation questions, similar to those asked in the Ramp-Up 
implementation study (Findsay, et. al, 2016): 

• Are there differences in the number or types of supplemental college -readiness 
supports (that is, programs or services) between Ramp-Up schools and comparison 
schools? 

• Are there differences between Ramp-Up schools and comparison schools in staff 
engagement in college-oriented activities (adoption of curriculum and technology, 
professional development, college-focused student interactions, and postsecondary 
planning)? 

• To what extent did schools implement the core components of Ramp-Up (struc- 
tural supports, professional development, curriculum delivery, curriculum content, 
and postsecondary planning tools) as intended by the program developer? 

The research questions were addressed through a randomized controlled trial in which 
half of participating schools were randomly assigned to begin implementing Ramp-Up 
during the 2014/15 school year and the other half continued their normal college readiness 
programming until 2015/16. See box 2 for a summary of the data and methods used in the 
study and appendix B for more detail. 
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Box 2. Data and methods 


Study design. This study was a randomized controlled trial. Fifty schools serving grades 10-12 (48 in Minnesota and 
2 in western Wisconsin) applied to participate in this study in return for receiving training, materials, and coaching 
for the University of Minnesota’s College Readiness Consortium Ramp-Up to Readiness™ program at no cost. The 
study team randomly assigned schools to implement Ramp-Up either during the 2014/15 school year (the Ramp-Up 
group) or during the 2015/16 school year (the comparison group) through a two-stage process. First, the study team 
grouped schools into seven blocks, each comprising schools that were similar either by location or by a score that 
combined a school’s rate of math proficiency with its rate of student eligibility for the federal school lunch program. 
Second, half the schools within each block were randomly assigned to one of the two conditions (see figure B1 in 
appendix B for details on the randomization procedure). The study team randomly assigned each of the two Wiscon- 
sin schools to different study conditions. One school in the comparison group dropped out of the study after schools 
were randomly assigned to groups, so the final analytic sample included 49 schools: 25 schools in the Ramp-Up 
group and 24 schools in the comparison group. 

Data collection. To address the research questions, the study team collected the following data: 

• School-level data from the Minnesota Department of Education and the Wisconsin Department of Public 
Instruction. 

• Extant student records from schools and school districts. 

• Responses to a survey administered in fall 2014 to students in grades 10-12 in both groups of schools and a 
follow-up survey administered to those same students in spring 2015. (Data on completion of the Free Applica- 
tion for Federal Student Aid [FAFSA] and submission of college applications are from these surveys.) 

• Responses to items on ACT Engage, a survey that measures student factors associated with academic success, 
such as student motivation and skills, social engagement, and self-regulation and that was administered to 
randomly selected students in grades 10-12 in the fall and spring (see appendix B for more details on the ACT 
Engage survey). Analyses focused on the goal striving and commitment to college scales from the survey. 

• Teachers’ responses to items on five online instructional logs (Ramp-Up schools only). 

• Responses to an online survey administered to staff in all schools in the fall and to staff in Ramp-Up schools 
only in the spring. 

• Extant documents from the consortium regarding Ramp-Up implementation. 

See table B2 in appendix B for a crosswalk of what data were used to address each research question. 

Immediate outcome measures The study team constructed the following outcome measures to address the confir- 
matory and exploratory research questions: 

• FAFSA completion. In spring 2015 students completed a survey that asked “Have you submitted the Free Appli- 
cation for Federal Student Aid (FAFSA) so far this school year?” A dichotomous variable was created, with 1 
representing having completed it and 0 representing not having completed it or not knowing. 

• Personal college readiness. Two scales from ACT Engage were used to measure students’ personal college 
readiness: goal striving and commitment to college. Goal striving is a composite of 10 items that measure the 
“strength of [a student’s] efforts to achieve [his or her] objectives and end goals” (ACT, 2012, p. 2). Sample 
items in this scale include “Once I set a goal, I do my best to achieve it” and “I bounce back after facing dis- 
appointment or failure" (ACT, 2012, p. 34). Commitment to college is a composite of 10 items that measure a 
student’s commitment to enrolling in and completing college (ACT, 2012). Sample items in this scale include “A 
college education will help me achieve my goals” and “I am committed to attend and finish college regardless of 
obstacles” (ACT, 2012, p. 33). 

• College applications. In spring 2015, students completed a survey that asked “How many applications, if any, 
have you submitted so far this school year?” A dichotomous variable was created, with 1 representing having 
submitted at least one college application and 0 representing having submitted no college applications. 

(continued) 
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Box 2. Data and methods (continued) 


Student-level sample sizes. The sample size for the four outcome measures ranged from 1,526 students for submit- 
ting at least one college application to 2,128 students for FAFSA completion. The sample size for the two measures 
of personal college readiness was 2,309 students. 

Magnitude of effects. The magnitude of the effects of Ramp-Up are presented as odds ratios for the dichotomous 
outcomes of likelihood of FAFSA completion and likelihood of submitting at least one college application and as 
Fledges’s g for the continuous outcomes of ACT goal striving and commitment to college scale scores. Odds ratio is 
a standardized metric for conveying the magnitude of a group difference for a binary outcome. Odds ratios can be 
interpreted as the change in odds of performing the action that would occur as a function of being in the intervention 
group rather than the control group. For example, an odds ratio of 1.3 for FAFSA completion would suggest that if 
the average student moved from a comparison school to a Ramp-Up school, the odds of completing the FAFSA would 
increase 30 percent. Fledges’s g is a standardized metric for conveying the magnitude of a group difference on a 
continuous outcome. Fledges’s g represents the difference in mean values between the two groups in standard devi- 
ation units for the comparison group. See appendix B for more information on how these effect sizes are calculated. 

See appendix B for more information about data sources, missing data, imputation methods, and outcome 
measures. 

Data analysis. Confirmatory and exploratory research questions were addressed using hierarchical linear models 
with students nested in schools. Student-level covariates (indicator variables for race/ethnicity, gender, eligibility 
for the federal school lunch program, Individualized Education Program status, English learner status, grade level, 
unweighted cumulative high school grade point average, and Minnesota Comprehensive Assessment or Wisconsin 
Concepts and Knowledge Examination math scores) and school-level covariates (indicator variables for treatment 
condition, randomization block membership, and a prior school-level measure of the dependent variable such as the 
percentage of grade 12 students submitting the Free Application for Federal Student Aid in 2013/14) were used to 
statistically control for other potential factors that may affect the outcomes of interest. To address questions about 
treatment contrast, the study team examined staff members’ responses to survey items about college-readiness 
supports, students’ and staff members’ responses to survey items about college-focused activities, and students’ 
responses to survey items about the five dimensions of readiness. 

To address questions about fidelity of implementation among Ramp-Up schools, the study team worked with 
the consortium to break down Ramp-Up’s logic model and implementation model into core components, subcom- 
ponents, and the types of evidence (or indicators) that would verify whether the subcomponents were fully imple- 
mented (coded 1) or partially implemented (coded between 0 and 1; see figure B2 in appendix B). The study team 
calculated the percentage of indicators present for each subcomponent and component per school (the fidelity 
indexes). The consortium established thresholds for categorizing fidelity indexes as indicative of excellent fidelity 
(90 percent or higher), adequate fidelity (60-89 percent), or inadequate fidelity (at or below 59 percent) during the 
first year of implementation. 

Baseline equivalence. The study team first examined the characteristics of schools at baseline. The differences 
between the two groups of schools ranged from 0.00 standard deviation unit (ACT composite score) to 0.41 stan- 
dard deviation unit (percentage of student population that is Native American). Flowever, none of the differences was 
statistically significant (see table Cl in appendix C). 
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What the study found 


After one year of implementation, Ramp-Up did not have an impact on the three imme- 
diate outcomes examined in this study: likelihood of completing the FAFSA, score on the 
ACT Engage goal striving scale, and score on the ACT Engage commitment to college 
scale. Nor did Ramp-Up have a statistically significant impact on students’ submission of 
at least one college application. No differences were found across the student subgroups 
examined. Results also suggested uneven implementation, with most, but not all, schools 
adequately implementing some components. 

After the first year of implementation, students in Ramp-Up schools and students in comparison 
schools were equally likely to complete the Free Application for Federal Student Aid and had similar 
levels of personal college readiness 

The analysis of students’ FAFSA completion and personal college readiness statistically 
controlled for other student- and school-level factors. Little difference was found between 
students in Ramp-Up schools and students in comparison schools (figure 2; see also table 
C2 in appendix C). 

Submission of the Free Application for Federal Student Aid. While the percentage of 
grade 12 students who reported completing the FAFSA was higher in Ramp-Up schools 
(58 percent) than in comparison schools (51 percent; see figure 2), the difference was 
not statistically significant after other student- and school-level factors were controlled 
for (odds ratio = 1.3, p = .292). If the study had included a larger sample of students and 
schools, the difference between the Ramp-Up and comparison school students could have 
been statistically significant. 

ACT Engage goal striving scale score. No differences were found in students’ ACT Engage 
goal striving scale scores between Ramp-Up schools (mean = 48, standard deviation = 8.4) 
and comparison schools (mean = 48, standard deviation = 7.9; Hedges’s g = -0.02, p = .8541- 

ACT Engage commitment to college scale score. No differences were found in students’ 
ACT Engage commitment to college scale scores between Ramp-Up schools (mean = 48, 
standard deviation = 9.9) and comparison schools (mean = 48, standard deviation = 9.6; 
Hedges’s g = -0.04, p = .957). 

After the first year of implementation, students in Ramp-Up schools and students in comparison 
schools were equally likely to submit at least one college application 

The percentage of grade 12 students who submitted at least one college application was 
higher in Ramp-Up schools (88 percent) than in comparison schools (82 percent), but the 
difference was not statistically significant once other student- and school-level factors were 
controlled for (odds ratio = 1.26, p = .477; see table C2 in appendix C). 4 The analysis of 
Ramp-Up’s impact on this outcome after one year of implementation is considered explor- 
atory because the program developer was unsure whether a single year of implementation 
would be sufficient to produce impacts on submission of college applications. 
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Figure 2. Students in Ramp-Up schools and students in comparison schools 
showed little difference in immediate outcomes, 2014/15 


Percent or percentile 



FAFSA completion 
(percent) 


Score on the ACT Engage Score on the ACT Engage 

goal striving scale commitment to college scale 

(percentile) (percentile) 


FAFSA is the Free Application for Federal Student Aid. 

Note: FAFSA completion is the average percentage of students who completed the application, after adjust- 
ment for covariates. The average scores for the two ACT Engage scales are average percentile scores after 
adjustment for covariates. None of the differences between Ramp-Up schools and comparison schools was 
statistically significant at the p < .05 level. Results for FAFSA completion are based on 2,128 grade 12 stu- 
dents in 43 schools. Results for ACT Engage goal striving and commitment to college scale scores are based 
on 2,039 students in 45 schools. For both analyses, schools that were included in the analytic samples did 
not differ significantly from schools that were not included in the sample on any baseline measures, including 
eligibility for the federal school lunch program, the percentage of English learner students, the percentage of 
students with an Individualized Education Program, percentages of racial/ethnic minority students, standard- 
ized test scores, graduation rates, or average ACT scores. 

Source: Authors' analysis of data from student surveys and ACT Engage surveys. 
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striving scale scores 
or in students’ 

ACT Engage 
commitment to 
college scale scores 
between Ramp- 
Up schools and 
comparison schools 


After the first year of implementation, Ramp-Up had no impacts on immediate and exploratory 
outcomes among subgroups of students 

The results show no impacts of Ramp-Up among students who scored in the upper third 
on the grade 8 state standardized math test, among students who scored in the middle 
third on the test, or among students eligible for the federal school lunch program (see 
tables C3, C4, C5 and figure Cl in appendix C). 5 

Qroup differences among students scoring in the upper third on the grade 8 state stan- 
dardized math achievement test. Among students who scored in the upper third on their 
state’s grade 8 math achievement test, there were no statistically significant differences 
between those in Ramp-Up schools and those in comparison schools in likelihood of com- 
pleting the FAFSA (odds ratio = 1.4, p = .208), scores on the ACT Engage goal striving 
scale (Hedges’s g = 0.06, p = .853), scores on the ACT Engage commitment to college scale 
(Hedges’s g = -0.16; p = .647), or likelihood of submitting at least one college application 
(odds ratio = 1.19, p = .799). 

Qroup differences among students scoring in the middle third on the grade 8 state 
standardized math test. For students who scored in the middle third on their state’s grade 
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8 math achievement test, there were no statistically significant differences between those 
in Ramp-Up schools and those in comparison schools in likelihood of completing the 
FAFSA (odds ratio = 1.2, p = .414), scores on the ACT Engage goal striving scale (Hedges’s 
g = 0.01, p = .529), scores on the ACT Engage commitment to college scale (Hedges’s g = 
-0.09, p = .285), or likelihood of submitting at least one college application (odds ratio = 
1.66, p = .308). 


Qroup differences among students eligible for the federal school lunch program. For 

students who were eligible for the federal school lunch program, there were no statisti- 
cally significant differences between those in Ramp-Up schools and those in comparison 
schools in likelihood of completing the FAFSA (odds ratio = 1.4, p = .365), scores on the 
ACT Engage goal striving scale (Hedges’s g = -0.13, p = .844), scores on the ACT Engage 
commitment to college scale (Hedges’s g = -0.18, p = 0.708), or likelihood of submitting at 
least one college application (odds ratio = 1.11, p = .821). 

Ramp-Up schools and comparison schools offered the same supplemental college-readiness supports, 
but staff in Ramp-Up schools showed more college-focused activity than did staff in comparison schools 

To demonstrate an impact, the Ramp-Up program would have to add college-readiness 
supports that were not already available in schools. The study team therefore examined 
whether college-readiness programming in Ramp-Up schools differed from that in the 
comparison schools. 

Basic college-readiness supports. Staff survey data collected in the fall of the 2014/15 
school year showed that Ramp-Up schools and comparison schools offered students similar 
supplemental college-readiness supports (for example, assistance with identifying colleges 
that match students’ interests and abilities and providing classes and workshops to prepare 
students for college entrance exams). The two groups of schools also showed similar rates 
of tracking students’ completion of key college enrollment actions (for example, college 
applications and college entrance exams; table 1). 

Students receiving assistance with enrollment actions. While staff in both groups of 
schools reported offering students assistance with college enrollment actions, a higher 
percentage of students received such assistance in Ramp-Up schools than in compar- 
ison schools (figure 3). Staff reported that a significantly higher percentage of students 
in Ramp-Up schools than of students in comparison schools received support filling out 
financial aid forms (p = .009), identifying scholarship opportunities (p = .033), and com- 
pleting scholarship applications (p < .022). A higher percentage of students in Ramp-Up 
schools may also have received assistance with completing college applications and plan- 
ning how to pay for college, but the average differences between the two groups of schools 
were not statistically significant (p = .051 for completing college applications and p = .06 
for planning how to pay for college). 

Engagement in college-oriented activity among staff. Survey data collected from staff and 
students indicate that staff in Ramp-Up schools were more likely than staff in comparison 
schools to engage in college-oriented activities in four domains: curriculum and technol- 
ogy, professional development, college-focused staff-student interactions, and postsecond- 
ary planning. Specifically, staff in Ramp-Up schools tended to provide more advanced 
course offerings and invest in technological platforms that track students’ progress toward 


For students who 
were eligible for the 
federal school lunch 
program, there 
were no statistically 
significant 
differences 
between those in 
Ramp-Up schools 
and those in 
comparison schools 
in likelihood of 
completing the 
FAFSA, scores on 
the ACT Engage 
goal striving 
scale, scores on 
the ACT Engage 
commitment to 
college scale, 
or likelihood of 
submitting at 
least one college 
application 


10 


Table 1. Staff in Ramp-Up schools and comparison schools reported offering 
the same basic college-readiness supports, fall 2014 (percent of staff member 
respondents) 


Types of support 

Ramp-Up 

schools 

Comparison 

schools 

Holding or participating in college fairs 

100 

100 

Consulting with college representatives 

96 

100 

Encouraging students to visit colleges 

100 

100 

Organizing college visits 

100 

96 

Offering programs that help students plan or prepare for college (such as Upward 
Bound, Advancement Via Individual Determination, and College Possible) 

96 

100 

Information sessions on searching for and applying to college 

92 

100 

Information sessions on paying for college 

88 

96 

Does school assist students with 

Identifying colleges that match students’ interests and abilities? 

100 

100 

Completing or reviewing college applications? 

100 

100 

Completing or reviewing the Free Application for Federal Student Aid? 

100 

96 

Identifying scholarship opportunities? 

100 

100 

Completing or reviewing scholarship applications? 

100 

96 

Classes or workshops to prepare for college admissions exams? 

100 

92 

Does school track students' completion of 

College applications? 

61 

58 

Free Application for Federal Student Aid? 

62 

45 

Scholarship applications? 

60 

69 

Completion of college admissions exam? 

60 

64 


Note: The sample consisted of 635 staff members from 25 Ramp-Up schools and 570 staff members from 
24 comparison schools. There were no statistically significant differences in any of these college-readiness 
supports between the two groups of schools. 

Source: Authors' analysis of staff survey data from fall 2014. 
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postsecondary goals, participate in professional development related to college readiness, 
emphasize college readiness in their interactions with students, and use college planning 
tools ( p < .05; figure 4). Furthermore, according to student survey respondents, staff in 
Ramp-Up schools showed greater emphasis on two of the five dimensions of college read- 
iness (admissions readiness and financial readiness) than did staff at comparison schools. 
Students’ observations of staff emphasis on academic readiness, career readiness, and per- 
sonal-social readiness did not differ between the two groups of schools (figure 5). 

Despite school staff members’ enhanced engagement in college-oriented activities in Ramp-Up 
schools, implementation of Ramp-Up components after the first year was inadequate for many 
schools 

Schools’ average implementation scores across all components indicate that 24 of 25 
Ramp-Up schools (96 percent) had average implementation scores in the adequate range 
(figure 6; see also table C6 in appendix C). However, 3 of the 25 implementing schools 
(12 percent) adequately implemented all five Ramp-Up components. The components 
most often found lacking in this study were the same as those found lacking in the 
Ramp-Up implementation study (Lindsay et al., 2016), specifically curriculum content and 
postsecondary planning tools. 


11 






Figure 3. According to staff responses, the percentage of students who received 
assistance with key enrollment actions was higher in Ramp-Up schools than in 
comparison schools, fall 2014 


Percent of students 



Filling out 
financial aid 
forms** 


Identifying 

scholarships* 


Completing 

scholarship 

applications* 


Completing 

college 

applications 


Planning for 
paying for 
college 


* Group differences were statistically significant at p < .05; ** group differences were statistically significant 
at p < .01. 

Note: The sample consisted of 287 staff members from 24 Ramp-Up schools and 232 staff members from 
23 comparison schools. Percentages are from the midpoint value in each response category, averaged across 
staff respondents within schools and then across schools for each group. 

Source: Authors' analysis of staff survey data from fall 2014. 


Figure 4. Staff in Ramp-Up schools were more likely than staff in comparison 
schools to engage in college-oriented activities, 2014/15 


Percent of indicators present 



Curriculum and Professional College-focused Postsecondary 

technology** development*** staff-student planning* 

interactions 8 * 


* Difference is significant at p < .05; ** difference is significant at p < .01; *** difference is significant at 

p < .001. 

Note: Percentages are based on responses from the student (5,913 in fall 2014 and 5,157 in spring 2015) 
and staff surveys (1,199 in fall 2014). Percentages were averaged across respondents within schools and 
then across schools for each group. 

Source: Authors' analysis of staff survey data from fall 2014, extant documents and data, and student survey 
data from fall 2014 and spring 2015. 
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Figure 5. Students in Ramp-Up schools observed significantly more staff support 
for two of the five dimensions of college readiness than did students in comparison 
schools, 2014/15 

Average score 

100 -i ■ Ramp-Up schools (n = 25) ■ Comparison schools ( n = 24) 



Academic Admissions* Career Financial** Personal-social 


* Significantly different at p < .05; ** significantly different at p < .01. 

Note: Based on responses from 5,913 students in fall 2014 and 5,157 students in spring 2015. 
Source; Authors' analysis of student survey data from fall 2014 and spring 2015. 


Figure 6. For Ramp-Up schools, overall implementation was excellent or adequate, 
but implementation for some schools on some components was inadequate, 
2014/15 


Percent of schools 
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Overall Structural Professional Curriculum Curriculum Planning 

fidelity supports development delivery content tools 


Components 


Note: Cutpoints for excellent, adequate, and inadequate implementation were set by the program developer. 
Source: Authors’ analyses of staff survey data, student survey data, and instructional log data from spring 2014. 
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Structural supports. One of the 25 Ramp-Up schools (4 percent) demonstrated excellent 
implementation, and 23 (92 percent) adequately established the structural supports needed 
for Ramp-Up to be successful (see figure B2 in appendix B for a list of structural supports). 
One school (4 percent) did not adequately develop the structural supports. 

Professional development. Of the 24 Ramp-Up schools with available data, 8 (32 percent) 
had professional development implementation scores in the excellent fidelity range, while 
the remaining 16 schools (64 percent) had scores in the adequate fidelity range. 

Curriculum delivery. All but one Ramp-Up school (96 percent) adequately delivered the 
Ramp-Up content. That is, the Ramp-Up advisors conducted most advisory sessions and 
workshops and had adequate access to Ramp-Up materials and resources. The remaining 
school inadequately delivered the curriculum to students. 

Curriculum content. Sixty-three percent of Ramp-Up schools conveyed the curriculum 
content — that is, content related to the five pillars of college readiness — at adequate levels. 
Thirty-seven percent of schools inadequately implemented this component. 

Postsecondary planning tools. The Ramp-Up component that the vast majority of schools 
failed to implement adequately was the development and use of college and career plan- 
ning tools. Only 12 percent of Ramp-Up schools implemented this component adequately, 
while the other 88 percent of schools inadequately implemented it. As in the Ramp-Up 
implementation study (Lindsay, et. al, 2016), the current study’s implementation scores on 
the postsecondary planning tools subcomponents showed that no schools were using plans 
as a means of keeping parents up-to-date on students’ progress toward college (see table C6 
in appendix C). 


Implications of the study findings 


Ramp-Up to Readiness is a schoolwide approach that aims to help middle and high schools 
improve students’ chances of college enrollment and success. The findings from this study 
may offer school district leaders, staff, and policymakers insight on the impacts they can 
expect after a single year of implementation for students in grades 10-12, how Ramp-Up 
complements other college-readiness approaches, and how well schools implemented 
Ramp-Up in the first year. 

This study suggests that Ramp-Up schools are unlikely to see immediate substantial 
impacts on student outcomes, such as FAFSA completion and ACT Engage goal setting 
and commitment to college scale scores, within a single year of implementation. Despite 
the belief of the University of Minnesota’s College Readiness Consortium, which devel- 
oped Ramp-Up, that the program would produce positive changes on these outcomes 
during the initial year, a single year may not be sufficient time for schools to fully imple- 
ment the program, and students may need to be exposed to more than a year of advisories 
and workshops for substantial impacts to occur. Exploratory findings also indicate that 
grade 12 students in Ramp-Up schools were no more likely than students in compari- 
son schools to submit at least one college application during the initial year of Ramp-Up 
implementation. The lack of difference may suggest that students require more than six 
to nine months of exposure to Ramp-Up activities for the impacts to occur. In addition, 
as described above, Ramp-Up schools may not have seen impacts on immediate outcomes 
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because of inadequate implementation of all core components or a lack of treatment corn 
trast (that is, since the schools all volunteered to participate in the study, comparison 
schools may already be placing additional emphasis on students’ academic readiness, career 
readiness, and personal-social readiness). 

This study found that Ramp-Up and comparison schools offer similar types of supplemen- 
tal college-readiness supports for students who self-identify as college bound. According to 
the consortium, a distinctive feature of Ramp-Up is its schoolwide approach, but given the 
evidence, it is unclear whether this feature is being actualized. For example, although more 
students in Ramp-Up schools than in comparison schools report receiving assistance with 
college enrollment actions, 49-59 percent of staff in schools in each group who responded 
to the fall staff survey were unable to estimate the percentage of students who received 
such assistance. 

Staff in Ramp-Up schools indicated that they engaged in more college-oriented activi- 
ties than did staff in comparison schools (see figure 4), but the additional engagement 
did not translate into differences on all five dimensions of college readiness between the 
two groups of schools. Statistically significant differences between students in Ramp-Up 
schools and students in comparison schools were found for only two of the five dimensions 
(admissions readiness and financial readiness; see figure 5). The lack of statistically signifi- 
cant differences on the other three dimensions — academic readiness, career readiness, and 
personal-social readiness — may be due to several features of the Ramp-Up implementa- 
tion. First, there was inadequate implementation of all core components. Second, there 
may have been a lack of program maturity in schools. It may take more than a single year 
to work out the implementation challenges. Third, there may have been a lack of program 
exposure among students. Students were exposed to only one year of Ramp-Up program- 
ming at most, whereas when fully implemented, they would have been exposed to it from 
grade 6. Finally, there may have been a lack of treatment contrast; comparison schools 
may also be placing additional emphasis on students’ academic readiness, career readiness, 
and personal-social readiness. Studies that examine Ramp-Up impacts after several years 
of implementation would likely provide a clearer picture of whether Ramp-Up activities are 
capable of producing the intended outcomes. 

For schools serving students in grades 10-12, implementation of Ramp-Up required making 
structural changes, supporting professional development for staff involved with Ramp-Up, 
establishing the process for conveying the Ramp-Up curriculum (establishing weekly advi- 
sory sessions and the five workshops), ensuring that all five dimensions of college readiness 
were conveyed through the advisory and workshop sessions, and establishing a college and 
career planning process. The findings indicate that some Ramp-Up schools fell short of 
adequate implementation of some core components (see figure 6). One possible implica- 
tion of this finding is that the consortium might provide schools with more guidance on 
implementing particular components, including curriculum content and postsecondary 
planning tools. Schools implementing the program may need to ensure that advisories 
and workshops cover all five dimensions of college readiness, that students and Ramp-Up 
advisors use postsecondary planning tools, and that parents are informed about students’ 
progress on actualizing their career plans. 
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Limitations of the study 


Several limitations should be kept in mind when interpreting the study findings. These 
limitations involve the size and integrity of the analytic samples, the outcome measures, 
inferences that can be drawn after a single year of program implementation, and the gen- 
eralizability of the findings. 

Size and integrity of the analytic samples 

The samples were problematic in two ways: the number of students per school was smaller 
than expected, leading to reduced ability to detect smaller effects, and for some outcomes 
the student-level response rates and attrition rates differed between Ramp-Up schools and 
comparison schools. 

The first problem involves the power of the statistical comparisons between groups — that 
is, whether the study included enough schools and students to detect the small effects 
that are often seen in studies using postsecondary outcomes. This impact study was orig- 
inally designed to detect effects as small as 5 percentage points for binary outcomes and 
0.17 standard deviation unit for continuous outcomes. However, given the recruitment of 
schools with fewer students, lower than expected response rates, and higher than expected 
attrition rates, the study could detect only larger effects — 15 percentage points for binary 
outcomes and 0.19 standard deviation for continuous outcomes — with adequate power 
(see tables B6 and B7 in appendix B for information on attrition and minimum detect- 
able effect sizes). For example, the power analysis for the actual sample sizes suggests that 
impacts greater than 19 percentage points on FAFSA completion would be detectable with 
95 percent confidence; however, the estimated impact was 4 percentage points, which did 
not meet the threshold. 

Response rates for fall and spring staff surveys were 50-70 percent, which some meth- 
odologists would consider barely acceptable (Mangione & Van Ness, 2009). Typically, 
such response rates would indicate that the findings based on these surveys (in this case, 
the findings on implementation fidelity and treatment contrasts) should be viewed with 
caution. However, the findings from completed staff surveys were consistent with those in 
the previous Ramp-Up study on implementation (Lindsay, et al, 2016). 

In some cases, differential response and attrition rates at the school and student levels may 
lead to skepticism over whether impact estimates can be attributed solely to Ramp-Up 
(see table B6 in appendix B). Differential response and attrition rates may mean that the 
intervention caused some potential respondents to leave the school or to not complete 
the outcome measures; the data would therefore fail to reflect their thoughts, feelings, 
or actions. Impact estimates for the following subgroup contrasts should be interpreted 
with caution because the samples did not meet What Works Clearinghouse standards for 
acceptable attrition: 

• For students in the upper third of the distribution on the grade 8 state standard- 
ized math test: Ramp-Up impact estimates of submission of at least one college 
application. 

• For students in the middle third of the distribution on the grade 8 state standard- 
ized math test: Ramp-Up impact estimates for FAFSA completion and scores on 
the ACT Engage goal striving and commitment to college scales. 


Given the 
recruitment of 
schools with fewer 
students, lower 
than expected 
response rates, 
and higher than 
expected attrition 
rates, the study 
could detect only 
larger effects 
-15 percentage 
points for binary 
outcomes and 0.19 
standard deviation 
for continuous 
outcomes-with 
adequate power 
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• For students eligible for the federal school lunch program: Ramp-Up impact esti- 
mates for FAFSA completion and submission of at least one college application. 

For the entire sample, attrition was high on the ACT Engage goal striving and commit- 
ment to college outcome measures as well (see table B6 in appendix B). However, the 
analytic samples from Ramp-Up and comparison schools showed baseline differences that 
were within the acceptable range used by the What Works Clearinghouse (Hedges’s g = 
-0.13 for goal striving and Hedges’s g = -0.16 for commitment to college). 

The outcome measures 

The study used students’ self-reports to measure whether they completed the FAFSA 
and submitted at least one college application because data could not be obtained from 
government agency databases, such as Minnesota’s state longitudinal data system or the 
National Student Clearinghouse, because of privacy concerns. Participating schools did 
not adequately track students’ transcript requests for college applications. Thus, these out- 
comes may have been influenced by student biases. The extent of this bias for students in 
Ramp-Up schools and comparison schools is unknown. 

Inferences that can be drawn after a single year of program implementation 

As noted in the Ramp-Up implementation (Lindsay, et. al, 2016), more than a single year 
may be needed to secure staff buy-in to the program. Staff too may need time to become 
more familiar with the curriculum and its delivery to carry out their advisor role with 
fidelity. Students may need multiple years of exposure to Ramp-Up’s curriculum to show 
impacts on immediate, intermediate, and long-term outcomes. This would be especially 
true for two of the key outcomes: FAFSA completion and submission of at least one college 
application. Future studies could benefit from examining the impacts of Ramp-Up follow- 
ing multiple years of implementation and student exposure to the curriculum. 

Generalizability of the findings 

Finally, findings from this study may not generalize to other types of students or other 
types of student populations. This study was conducted in high schools in Minnesota and 
western Wisconsin that volunteered to participate. The schools and student populations 
that participated in the study may differ from those in other areas of the United States; 
thus similar studies of Ramp-Up in other settings could yield different findings. 


Findings from 
this study may 
not generalize 
to other types of 
students or other 
types of student 
populations 
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Appendix A. Ramp-up program history 


This appendix provides a detailed description of the history of the Ramp-Up to Readi- 
ness program (Ramp-Up), including a timeline of its implementation. The appendix also 
includes information on how Ramp-Up is implemented in schools and the activities that 
compose the program. 

Program history and timeline 

Since 2010 the University of Minnesota’s College Readiness Consortium has implemented 
and tested Ramp-Up in 95 schools located throughout Minnesota and in 2 schools in 
western Wisconsin. The timeline of the program’s development is below: 


2007-09 

The consortium convened an advisory group of researchers, guidance counselors, 
school administrators, and higher education administrators to develop a program 
to increase the number and diversity of students who graduate high school ready to 
succeed in postsecondary settings. They identified the five dimensions of college 
readiness from research literature, which became the foundation of Ramp-Up. 

2009-10 

The consortium partnered with Minnesota's Center for Applied Research on Educational 
Improvement to create activities, curriculum, an implementation timeline, student 
planning tools, resources, and a technology platform. The advisory group offered 

refinements to these elements. 

2010-12 

Ramp-Up was piloted in 11 secondary schools during the 2010/11 school year. 

Curriculum and planning tools were refined based on implementation experience. 

2012/13 
school year 

Ramp-Up was piloted again in 41 secondary schools in Minnesota. Feedback from 
school implementation teams helped further refine Ramp-Up implementation process, 
resources, and activities. 

2013/14 
school year 

Regional Educational Laboratory (REL) Midwest examined implementation of Ramp-Up in 
a sample of 20 schools, 10 of which implemented Ramp-Up immediately (2013/14) and 
10 of which delayed implementation by one year. 

2014/15 
school year 

REL Midwest studied immediate impacts with a new sample of 49 schools (47 in 
Minnesota and 2 in western Wisconsin). Schools were randomly assigned to implement 
Ramp-Up immediately (2014/15) or delay implementation for one year (2015/16). 


Program resources and activities 

Ramp-Up focuses on students in grades 6-12 (this study looked at grades 10-12). Schools 
adopting Ramp-Up teach a set of 28-30 minute lessons (called group advisory sessions) 
to groups of students (the size of which varies by school) at each grade level once a week 
and five class period-long workshops throughout the year. Each of the 28 lessons occurs 
during students’ homeroom periods and is typically led by homeroom teachers (referred to 
as Ramp-Up advisors). Topics covered in the curriculum address one or more of the five 
dimensions of college readiness. The five workshops cover new content related to one of 
the five dimensions of readiness and involve more student-advisor interaction. Ramp-Up 
advisors receive detailed plans for group advisory sessions and workshops, along with a 
general description of the curriculum in a grade-specific advisor guide. Ramp-Up coordina- 
tors also provide the advisors with implementation support and supplemental materials to 
share with students. 

The Ramp-Up lessons and workshops connect to elements in students’ postsecondary 
plans and their readiness rubric. In their postsecondary plan — which is completed early 
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in the sequence of advisory sessions — students describe their college and careers of inter- 
est, their planned coursework each year, and their extracurricular activities. Students use 
the readiness rubric to measure their progress on their plans. Students update the post- 
secondary plan once a year and update the readiness rubric three times a year. These tools 
are also shared with parents. 

To guide implementation, Ramp-Up schools assemble a leadership team consisting of an 
administrator, a teacher, and a school counselor and appoint a Ramp-Up coordinator to 
oversee implementation. Staff members in Ramp-Up high schools learn how to implement 
the program by participating in the following professional development activities: 

• A full-day planning session facilitated by the consortium for the Ramp-Up leader- 
ship teams from each school prior to the beginning of the school year. 

• A half-day training session facilitated by the consortium for the Ramp-Up coordi- 
nators from each school prior to the beginning of the school year. 

• A four-hour training session for school staff in each Ramp-Up school, facilitated 
by the school’s Ramp-Up leadership team and coordinator at the beginning of 
the school year, to introduce the Ramp-Up program, curriculum, and key college- 
readiness information. 

• A monthly 20-minute training session at a faculty meeting to preview specific 
Ramp-Up activities, which Ramp-Up advisors will lead throughout the month. 

The program developer provides materials, slides, and specific guidelines for schools to use 
during the school-led professional development activities. 
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Appendix B. Methods 


This appendix details the methods used in the study, including how schools were selected 
to participate, types of data used, how data were coded and cleaned, and what analytic 
methods were used to address each research question. 

Selecting schools to participate 

The University of Minnesota’s College Readiness Consortium conducted information ses- 
sions about Ramp-Up to Readiness™ (Ramp-Up) throughout Minnesota and in western 
sections of Wisconsin during the 2013/14 school year. At the end of these presentations, 
schools were invited to complete an application to implement the Ramp-Up program 
without cost. The 50 schools that applied were randomly assigned to conditions through 
a two-step process. First, schools were placed into one of seven groupings — or random 
assignment blocks — based on state, district, and rank of their index score (which consisted 
of the sum of the percentage of students eligible for the federal school lunch program and 
the percentage of students who scored proficient in math during the 2013/14 school year). 
Second, half of the schools in each block were randomly selected to implement Ramp-Up 
during the 2014/15 school year (Ramp-Up schools), and the other half were randomly 
selected to delay implementing Ramp-Up until following year (comparison schools; figure 
Bl). Regional Educational Laboratory (REL) Midwest paid $1,500 to each comparison 
school as compensation for delaying implementation of Ramp-Up and assisting with data 
collection. One comparison school was unable to provide the necessary data and was ulti- 
mately dropped from the study. 

A total of 25 Ramp-Up schools and 24 comparison schools were included in the analysis. 
Extant data were collected on 15,314 students: 7,574 students in Ramp-Up schools and 
7,740 students in comparison schools (table Bl). For Ramp-Up schools the average number 
of students in grades 10-12 was 687, with a range from approximately 50 to 1,400. For 
comparison schools the average number of students in grades 10-12 was 544, with a range 
from approximately 45 to 1,600. 6 

Description of data 

To address the research questions underlying this study, the study team collected different 
types of quantitative and qualitative data (table B2). 

Extant data. The following student-level records were requested from the 49 schools par- 
ticipating in the study: 

• Grade level. This study used data for students in grades 10-12 during the 2014/15 
school year. Students with missing data for grade level were excluded from the 
study. 

• Demographic characteristics. Measures included race/ethnicity, gender, eligibility for 
the federal school lunch program (a proxy for economic disadvantage), having an 
Individualized Education Program, and being an English learner student. The study 
team assigned students with missing race/ethnicity data to the modal category of 
White (82 percent of students in the sample with data were White). Students with 
missing gender data were randomly assigned to be either male or female. When 
students were missing data on eligibility for the federal school lunch program, the 
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Figure Bl. Process used to randomly assign schools to conditions 
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Table Bl. Number of students in grades 10-12 and total sample size by treatment 
condition, fall 2014 


1 Treatment condition 

Grade 10 

Grade 11 

Grade 12 

Total | 

Ramp-Up schools 

2,536 

2,555 

2,483 

7,574 

Comparison schools 

2,679 

2,493 

2,568 

7,740 

Total 

5,215 

5,048 

5,051 

15,314 


Source: Authors' calculations based on data obtained from participating schools. 


study team imputed the percentage of students in their school who qualified for the 
school lunch program. Finally, when students were missing data on having an Indi- 
vidualized Education Program or being an English learner student, the study team 
assumed it was because they did not meet the criteria for either status and assigned 
a 0 for each. 7 Whenever missing data were imputed, the study team created indi- 
cator variables to specify which students had missing data on particular variables. 

• Test scores on state accountability assessments. Students in Minnesota are required 
to take the Minnesota Comprehensive Assessment series II (MCA-II) in grades 
3-8. Grade 8 MCA4I math, reading, and science scores were collected for stu- 
dents in the sample (those in grades 10-12 in the 2014/15 school year). The study 
team chose to use students’ MCA-II math scores as covariates in the analyses. It 
first converted MCA-II math scores to z scores and then imputed a 0 (the mean) 
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Table B2. Data used to address each research question 


Research questions 

Data sources 

Confirmatory research questions 

What is the impact of Ramp-Up on the likelihood of 
grade 12 students completing the Free Application 
for Federal Student Aid (FAFSA)? 


Student-level extant data from schools 

School-level extant data from schools 

Minnesota Department of Education data 

U.S. Department of Education financial aid data 
Wisconsin Department of Public Instruction data 
Student survey data 

What is the impact of Ramp-Up on grade 10, 11, and 
12 students’ scores on the ACT Engage goal striving 
and commitment to college scales? 


Student-level extant data from schools 

School-level extant data from schools 

Minnesota Department of Education data 

Wisconsin Department of Public Instruction data 
ACT Engage survey data 

Exploratory research questions 

What is the impact of Ramp-Up on the likelihood of a 
student in grade 12 submitting at least one college 
application? 


Student-level extant data from schools 

School-level extant data from schools 

Minnesota Department of Education data 

Wisconsin Department of Public Instruction data 
Student survey data 

What is the impact of Ramp-Up on the likelihood of 
completing the FAFSA, ACT Engage goal striving and 
college commitment scale scores, and the likelihood 
of submitting at least one college application for 
students who scored in the upper third on the grade 

8 state standardized test, students who scored in 
the middle third on the grade 8 state standardized 
test, and students who are eligible for the federal 
school lunch program? 


Student-level extant data from schools 

School-level extant data from schools 

Minnesota Department of Education data 

Wisconsin Department of Public Instruction data 
U.S. Department of Education financial aid data 

ACT Engage survey data 

Student survey data 

Implementation research questions 

Are there differences in the number or types of 
supplemental college-readiness supports (that is, 
programs or services) between Ramp-Up schools 
and comparison schools? 


Student-level extant data 

Staff survey data 

Student survey data 

Are there differences between Ramp-Up schools 
and comparison schools in staff engagement in 
college-oriented activities (adoption of curriculum 
and technology, professional development, college- 
focused student interactions, and postsecondary 
planning)? 


Student-level extant data 

Staff survey data 

Student survey data 

To what extent did schools implement the core 
components of Ramp-Up (structural supports, 
professional development, curriculum delivery, 
curriculum content, and postsecondary planning 
tools) as intended by the program developer? 


Implementation data from program developer 

Staff survey data 

Student survey data 

Staff instructional log data 

Source: Authors' study plan. 


for all missing scores. Finally, the study team created an indicator variable to 
specify which students were missing MCA-II math scores. The same process was 
used with Wisconsin students’ grade 8 math scores on the Wisconsin Concepts 
and Knowledge Examination. 

• High school grade point average. Cumulative high school unweighted grade point 
average as of the fall of the 2014/15 school year was collected for all students in 
the study. When students were missing values for grade point average, the study 
team imputed the sample mean and created an indicator variable to specify which 
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students were missing data on grade point average. In addition, some schools 
submitted weighted grade point averages. Because the study team did not have 
sufficient information to be able to convert weighted grade point averages into 
unweighted grade point averages, a ceiling was imposed at 4.0. 

• Minnesota Automated Reporting Student System number and Wisconsin Student 
Number. Schools provided the unique identification number assigned to each 
student by their state, and these numbers were used to connect studentdevel 
extant data to student survey data. In several cases, schools did not use the state 
identification numbers as unique identifiers for students. The study team asked 
those schools to create their own unique identifier for each student and keep a 
crosswalk that could link students’ identifiers with their identities. 

The following schoohlevel extant data were collected from schools: 

• Number of students in grades 10, 11, and 12 in 2013/14- Schools provided data on the 
number of students in grades 10, 11, and 12 during the 2013/14 school year. These 
data were used as the denominator when calculating the percentage of students 
who completed the Free Application for Federal Student Aid (FAFSA) in 2013/14- 

The following schoohlevel extant data were obtained from publicly available data files 
maintained by the Minnesota Department of Education and the Wisconsin Department 
of Public Instruction: 

• Number of students in grades 10, 11, and 12 in 2013/14 ■ If schools did not report 
the number of students in grades 10, 11, and 12 during the 2013/14 school year, 
the study team obtained these data from data files maintained by the Minnesota 
Department of Education and Wisconsin Department of Public Instruction. 

• Percentage of students eligible for the federal school lunch program. For students 
missing data on eligibility for the federal school lunch program, the study team 
imputed the percentage of students in their school who were eligible. 

The following schoohlevel extant data were collected from the U.S. Department of Educa- 
tion’s Federal Student Aid website (https://studentaid.ed.gov/sa/about/data-center/student/ 
application-volume/fafsa-completion-high-school), which publishes data on FAFSA com- 
pletion by high school: 

• Number of grade 12 students who completed the FAFSA in 2013/14 by high school. 
Only high schools with five or more students who completed the FAFSA are 
included in the database. For the high schools that were included, the study team 
calculated the percentage of students in each school who completed the FAFSA 
in 2013/14 by dividing the number of FAFSAs submitted in 2013/14 by the number 
of grade 12 students in 2013/14. 

Student survey. In fall 2014 and spring 2015, information from students in Ramp-Up 
schools and comparison schools was gathered using a survey. A random sample of 30 grade 
10 students per school, a random sample of 30 grade 11 students per school, and all 5,051 
grade 12 students were selected to take the online survey. 8 The 10-15 minute survey was 
designed to help the study team better understand schools’ fidelity of implementation and 
the contrast between the college-readiness supports in schools implementing Ramp-Up 
and the comparison schools. The spring survey also asked grade 12 students about FAFSA 
completion and college applications. Students were asked “Have you submitted the Free 
Application for Federal Student Aid (FAFSA) so far this school year?” Four response 
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options (“Yes,” “No, but I plan to submit the FAFSA by the end of the summer, “No, I 
do not plan to submit the FAFSA,” and “I don’t know”) were recoded into a dichoto' 
mous variable in which 0 represents not having submitted the FAFSA yet or not knowing 
and 1 represents having completed the FAFSA. In addition, students were asked “How 
many college applications, if any, have you submitted so far this school year?” Five response 
options (none, one, two or three, four or five, and more than five) were recoded into a 
dichotomous variable in which 0 represents having submitted no college applications and 
1 represents having submitted at least one college application. 

Overall, 78 percent of students completed the fall survey (5,913 of 7,541 eligible students) 
and 68 percent of students completed the spring survey (5,517 of 7,558 eligible students). In 
Ramp-Up schools, 86 percent of students completed the fall survey (3,019 of 3,516 eligible 
students), and 79 percent of students completed the spring survey (2,743 of 3,494 eligible 
students). In comparison schools, 72 percent of students completed the fall survey (2,894 
of 4,025 eligible students), and 59 percent of students completed the spring survey (2,414 of 
4,064 eligible students). The average schooUevel response rate for Ramp-Up schools was 
83 percent for the fall survey and 74 percent for the spring survey. The average school-level 
response rate for comparison schools was 84 percent for the fall survey and 64 percent for 
the spring survey (table B3). 

ACT Engage survey. In fall 2014 and spring 2015, students in Ramp-Up schools and 
comparison schools completed the ACT Engage survey for grades 10-12. The study team 
requested that the same 30 randomly selected grade 10 and grade 11 students from each 
school invited to complete the student survey also complete the ACT Engage survey, and 
the study team randomly selected 30 grade 12 students from each school to complete the 
ACT Engage survey as well. 9 A total of 4,317 students were randomly selected to complete 
the ACT Engage survey. 10 

Overall, 79 percent of eligible students (3,406) completed the ACT Engage survey in 
the fall, and 62 percent (2,691) completed it again in the spring. In Ramp-Up schools, 


Table B3. Student survey and ACT Engage survey response rates for fall 2014 and spring 2015, by 
treatment condition 


Survey time 
and treatment 
condition 


Student survey 



ACT Engage survey 


Number of 
respondents 

Percentage 
of eligible 
respondents 
who completed 
the survey 

Average school- 
level response 
rate 

(percent) 

Number of 
respondents 

Percentage 
of eligible 
respondents 
who completed 
the survey 

Average school- 
level response 
rate 

(percent) 

Fall 2014 

Ramp-Up 

3,019 

86 

83 

1,606 

76 

77 

Comparison 

2,894 

72 

84 

1,800 

81 

82 

Total 

5,913 

78 

84 

3,406 

79 

80 

Spring 2015 

Ramp-Up 

2,743 

79 

74 

1,327 

63 

63 

Comparison 

2,414 

59 

64 

1,364 

61 

61 

Total 

5,157 

68 

69 

2,691 

62 

62 


Note: Based on responses from students in 25 Ramp-Up schools and 24 comparison schools that administered the student survey and 
the ACT Engage survey. 

Source; Authors' calculations based on response counts from the student survey and ACT Engage survey. 
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76 percent of students completed the survey in the fall (1,606 of 2,105 eligible students), 
and 63 percent of students completed it in the spring (1,327 of 2,105 eligible students). In 
comparison schools, 81 percent of students completed the survey in the fall (1,800 of 2,212 
eligible students), and 61 percent of students completed it in the spring (1,364 of 2,212 eli- 
gible students). The average school-level response rate for Ramp-Up schools was 77 percent 
for the fall survey and 63 percent for the spring survey. The average school-level response 
rate for comparison schools was 82 percent for the fall survey and 61 percent for the spring 
survey (see table B3). 

This sample size corresponds to a minimal detectable effect size of less than 0.10 in an 
analysis of the relationship between the ACT Engage scale scores and measures of imple- 
mentation fidelity. The ACT Engage survey is an online questionnaire that measures 
student factors associated with academic success, such as students’ motivation and skills, 
their social engagement, and their self-regulation. The grade 10-12 version of ACT Engage 
has 108 items and 10 scales in three domains and requires 15-20 minutes to complete. 
Analysis for this study focused on 2 of the 10 scales — goal striving and commitment to 
college — which the program developer considers measures of personal college readiness. 11 

The goal striving scale consists of 10 items that measure the “strength of [a student’s] efforts 
to achieve [his or her] objectives and end goals” (ACT, 2012, p. 2). ACT does not publish 
all items on the ACT Engage survey but does provide sample items for each scale. Sample 
items in this scale include “Once I set a goal, I do my best to achieve it” and “I bounce 
back after facing disappointment or failure” (ACT, 2012, p. 34). ACT states that this scale 
has good internal consistency reliability (alpha = .87). 

The commitment to college scale has 10 items that measure a student’s commitment to 
enrolling in and completing college. ACT states that this scale has good internal con- 
sistency reliability as well (alpha = .89; ACT, 2012). Sample items in this scale include 
“A college education will help me achieve my goals” and “I am committed to attend and 
finish college regardless of obstacles” (ACT, 2012, p. 33). Students’ scale scores on the two 
measures show a correlation of r = .60 (ACT, 2012). 

There is some predictive validity information available on the two scales as well. For example, 
the scales have a moderate correlation with high school grade point average (r = .3 for the 
goal striving scale and r = .4 for the commitment to college scale; ACT, 2012) and with 
college grade point average (r = .3 for both scales; Peterson, Casillas, & Robbins, 2006). In 
addition, the commitment to college scale predicts retention in two- and four-year colleges 
for the first year after institutional characteristics, student demographics, and prior academic 
achievement are controlled for (Robbins, Allen, Casillas, Peterson, & Le, 2006). 12 Specif- 
ically, a one standard deviation increase in commitment to college was associated with a 
19 percent increase in the odds of persisting after the first year at a four-year college. 

Staff surveys. In fall 2014, all staff at Ramp-Up schools and comparison schools were 
invited to complete a 20-30 minute online survey. The survey consisted of 26 items 
that inquired about the respondents’ personal beliefs about their role in helping students 
prepare for and apply to college, the types of college-readiness supports that the school 
makes available to students, the types of assistance that students receive from school staff, 
and the respondents’ perspectives on students’ readiness to succeed in a postsecondary 
institution. For Ramp-Up schools, the overall response rate was 62 percent (624 of 1,003 
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eligible staff members), and the average schooUevel response rate was 64 percent (ranging 
from 25 percent to 83 percent; table B4). For comparison schools, the overall response rate 
was 57 percent (575 of 1,009 eligible staff members), and the average schooUevel response 
rate was 56 percent (ranging from 0 percent to 91 percent). 

In April and May 2015, staff at Ramp-Up schools were asked to complete a second online 
survey. This 87-item survey asked staff about their perceptions of the strengths and weak- 
nesses of the Ramp-Up program’s curriculum, tools, and professional development. It also 
gathered information about whether school staff implemented the intervention as intended. 
Surveys were administered to members of schools’ Ramp-Up leadership team, the Ramp-Up 
coordinator, and any teachers in grades 10-12 who had a role in delivering Ramp-Up to 
students. The surveys included questions with scaled responses, as well as two open-ended 
questions asking about the strengths and the weaknesses of the Ramp-Up program. Respon- 
dents received a $25 gift card to Amazon.com for participating in the survey. The overall 
response rate was 50 percent (502 of 1,003 eligible staff members), and the average school- 
level response rate was 55 percent (ranging from 31 percent to 83 percent; see table B4). 

Instructional logs. Ramp-Up advisors were invited several times throughout the year 
to complete instructional logs after each workshop. In four schools, staff did not com- 
plete any instructional logs. Overall, 312 logs were completed for 1,808 workshops, for an 
average school-level response rate of 17 percent. The study team had intended to use the 
instructional log data to address an implementation-related research question. Because the 
response rates were so low, the study team opted to drop the research question. However, 
because instructional logs were completed by three or more Ramp-Up advisors in each 
school, the information listed about the topics covered and student attendance at work- 
shops was used in the calculation of implementation scores. 

Missing extant data. Overall, the rates of missing extant data varied. The missing data 
rates for demographic variables were lower, while the rates for academic achievement vari- 
ables were higher. Data were missing from the sample when the variable requested was not 
submitted by a school (table B5). 


Table B4. Staff survey response rates, by treatment condition, 2014/15 


Survey time and 
treatment condition 


Staff survey 


Number of respondents 

Percentage of eligible 
respondents who 
completed the survey 

Average school-level 
response rate 
(percent) 

Fall 2014 

Ramp-Up 

624 

62 

64 

Comparison 

575 

57 

56 

Total 

1,199 

60 

60 

Spring 2015 

Ramp-Up 

502 

50 

55 

Comparison 

na 

na 

na 

Total 

na 

na 

na 


na is not applicable because staff in comparison schools completed the fall surveys only. 

Note: Based on responses from staff in 25 Ramp-Up and 25 comparison schools that administered staff 
surveys. 

Source: Authors' calculations based on response counts from the staff survey. 
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Table B5. Rates of missing data in the analytic sample for extant variables by 
treatment condition, 2014/15 


Ramp-Up schools 

Comparison schools 


Variable 

Number of 
schools 

Number of 
students 

Percentage 

missing 

Number of 
schools 

Number of Percentage 
students missing 

Gender 

25 

155 

2 

24 

3 

0 

Race/ethnicity 

25 

0 

0 

23 

122 

2 

Eligibility for the federal school 
lunch program 

24 

1,371 

18 

24 

399 

5 

Individualized Education Program 

status 

25 

0 

0 

24 

0 

0 

English learner status 

25 

0 

0 

24 

0 

0 

Grade 8 state standardized math 

test score 3 

23 

2,720 

36 

22 

2,796 

36 

Cumulative unweighted grade 
point average (on a scale of 0-4) 

24 

203 

3 

23 

192 

3 


Note: The rates of missing data are based on the total number of students in grades 10-12 in Ramp-Up 
schools (n = 7,740) and comparison schools (n = 7,574). 

a. State standardized test refers to the Minnesota Comprehensive Assessment or the Wisconsin Concepts 
and Knowledge Examination. 

Source: Authors' calculations based on data from participating schools. 


Approval for data collection 

All study materials were reviewed and approved by American Institutes for Research’s Institu- 
tional Review Board. The study team also applied for and obtained clearance from the Office of 
Management and Budget to collect the data (OMB NOA 1850-0907). The study team adhered 
to all local policies and processes for securing consent from parents and school staff members. 

Process for coding and cleaning data 

Some data had to be converted into analyzable forms to answer questions about program 
impact and implementation. The process for this conversion, coding, and cleaning is 
described here. 

Extant data. Extant data were used to describe the context for schools participating in 
the study and to control for baseline student and school characteristics for all impact 
analyses. Various data-cleaning procedures were conducted, such as checking that data 
values fell within acceptable ranges, identifying duplicate cases of students, assessing the 
amount of missing data for individual variables, and examining the distribution of vari- 
ables. Values that were out of range were declared missing. No duplicate cases of student 
records were found. In many cases, new variables were calculated based on the provided 
data (for example, an indicator of enrolling in advanced coursework was calculated based 
on enrollment in different levels of coursework). 

Student survey, staff survey, and instructional log data. Student survey data were used 
to create four self-reported outcome variables for the impact analyses: FAFSA completion, 
ACT Engage goal striving scale score, ACT Engage commitment to college scale score, 
and submitting at least one college application). Student survey data were also used to 
assess fidelity of implementation and to describe the treatment contrast. 
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Staff survey responses were recoded such that responses reflected the extent to which the 
Ramp-Up model was implemented as intended by the program developer. These codes 
ranged from 0 (not at all indicative of correct implementation of the Ramp-Up model) to 1 
(definitely indicative of correct implementation of the Ramp-Up model). 

Instructional log data were used in calculation of implementation scores. Specifically, logs 
provided an indication of whether Ramp-Up advisors were covering the topics that were 
meant to be covered during the workshops (an indicator of the workshop subcomponent 
for curriculum delivery component). After the final log was completed in all Ramp-Up 
schools, the study team averaged the numeric codes across all logs for each teacher. 
School-level scores on each indicator are the average score of the indicator taken across 
respondents in that school. 

Analytic methods used to address research questions 

Following conversion of data into analyzable forms, the study team performed the preliminary 
analyses, the confirmatory and exploratory analyses, and analysis of program implementation. 

Preliminary analyses. Prior to conducting analyses to address the confirmatory, explor- 
atory, and implementation research questions, the study team conducted two preliminary 
sets of analyses. First, they examined attrition rates in both sets of schools (the percentage 
of students who were present in the school in October but not in May; table B6). Second, 


Table B6. Attrition rates for full samples and subsamples on all outcomes 


School-level attrition 

Student-level attrition 

Outcome 

Ramp-Up Comparison 
schools schools 

Difference 

Ramp-Up 

schools 

Comparison 

schools 

Difference 

Full sample 

Completed Free Application for Federal Student Aid 

0.16 

0.12 

0.04 

0 

0 

0 

Submitted at least one college application 

0.32 

0.24 

0.08 

0 

0 

0 

ACT Engage goal striving scale score 

0.12 

0.08 

0.04 

0.46 

0.38 

0.08 

ACT Engage commitment to college scale score 

0.12 

0.08 

0.04 

0.46 

0.38 

0.08 

Scored in the upper third on the grade 8 state standardized math test 8 

Completed Free Application for Federal Student Aid 

0.20 

0.24 

0.04 

0 

0 

0 

Submitted at least one college application 

0.44 

0.52 

0.08 

0 

0 

0 

ACT Engage goal striving scale score 

0.12 

0.12 

0 

0.35 

0.36 

0.01 

ACT Engage commitment to college scale score 

0.12 

0.12 

0 

0.35 

0.36 

0.01 

Scored in the middle third on the grade 8 state standardized math test 3 

Completed Free Application for Federal Student Aid 

0.12 

0.24 

0.12 

0 

0 

0 

Submitted at least one college application 

0.44 

0.48 

0.04 

0 

0 

0 

ACT Engage goal striving scale score 

0.32 

0.16 

0.16 

0.22 

0.27 

0.05 

ACT Engage commitment to college scale score 

0.32 

0.16 

0.16 

0.22 

0.27 

0.05 

Eligible for the federal school lunch program 

Completed Free Application for Federal Student Aid 

0.08 

0.2 

0.12 

0 

0 

0 

Submitted at least one college application 

0.40 

0.28 

0.12 

0 

0 

0 

ACT Engage goal striving scale score 

0.16 

0.08 

0.08 

0.55 

0.43 

0.12 

ACT Engage commitment to college scale score 

0.16 

0.08 

0.08 

0.55 

0.43 

0.12 

a. State standardized test refers to the Minnesota Comprehensive Assessment or the Wisconsin Concepts and Knowledge Examination. 

Source: Authors' calculations based on extant student data and student surveys from fall 2014 and spring 2015. 
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they examined the balance between the two groups of schools at baseline, determining 
whether Ramp-Up schools were equivalent to comparison schools on school-level student 
academic performance (schoolwide performance on the grade 8 state standardized math 
test), school size (number of students in grades 10, 11, and 12), and student demographic 
characteristics (percentage of students eligible for the federal school lunch program, per- 
centage of students who are English learner students, and percentage of students from a 
racial/ethnic minority group). 

Confirmatory and exploratory analyses. Confirmatory analyses assessed the effects of the 
Ramp-Up program on FAFSA completion and ACT Engage goal striving and commitment 
to college scale scores (see table B7 for power calculations). Exploratory analyses assessed 
the effects of the Ramp-Up program on students’ submission of college applications. Four 
intent-to-treat impact models were estimated using a two-level model with students nested 
within schools. All models assumed a constant treatment effect across blocks (see figure B1 


Table B7. Power calculations 

Measure 

Number of 
students 

Number 
of schools 

Effect size 

Minimal 
detectable 
effect size 

Full sample 

Completed Free Application for Federal Student Aid 

2,128 

43 

7 percentage 
points 

15 percentage 
points 

Submitted at least one college application 

1,526 

36 

6 percentage 
points 

14 percentage 
points 

ACT Engage goal striving scale score 

2,039 

45 

.04 

.19 

ACT Engage commitment to college scale score 

2,039 

45 

.02 

.19 

Scored in the upper third on the grade 8 state standardized math test 

Completed Free Application for Federal Student Aid 

445 

30 

4 percentage 
points 

18 percentage 
points 

Submitted at least one college application 

297 

26 

0 percentage 
points 

18 percentage 
points 

ACT Engage goal striving scale score 

390 

36 

.09 

.22 

ACT Engage commitment to college scale score 

390 

36 

.01 

.22 

Scored in the middle third on the grade 8 state standardized math test 

Completed Free Application for Federal Student Aid 

534 

34 

2 percentage 
points 

19 percentage 
points 

Submitted at least one college application 

374 

27 

7 percentage 
points 

15 percentage 
points 

ACT Engage goal striving scale score 

578 

40 

.16 

.21 

ACT Engage commitment to college scale score 

578 

40 

.06 

.21 

Eligible for federal school lunch program 

Completed Free Application for Federal Student Aid 

515 

42 

3 percentage 
points 

22 percentage 
points 

Submitted at least one college application 

365 

33 

2 percentage 
points 

19 percentage 
points 

ACT Engage goal striving scale score 

636 

44 

.18 

.20 

ACT Engage commitment to college scale score 

636 

44 

.13 

.20 


Note: Effect sizes for dichotomous outcomes are the difference in percentage points between students in 
Ramp-Up schools and students in comparison schools. Effect sizes for continuous outcomes are standardized 
mean differences (Hedges's g). Minimum detectable effect sizes for each outcome are based on sample sizes 
from the study, with alpha set at .05 and power set at .80. 

Source: Authors’ calculations based on data from the analytic sample. 
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for blocks) and include block at the school level. The treatment condition indicator, which 
specifies whether a student attended a Ramp-Up school or a comparison school, is includ- 
ed at the school level. For dichotomous variables, a logit link function is used to transform 
the dependent variable into the odds of achieving a particular outcome. 


The general model for all outcomes was: 
Level 1 model: Students-within-schools 


P 

y k = Poi + X + 

P= 1 

where i is students in school; (i = 1, ..., I), j is high schools (j = 1,..., J), v] r is the outcome 
for student i in high school j, a pi . is the pth student characteristic for student i in high 
school j, and e is a random error term for student i in high school j. 


Level 2 model: Schools 


@0j boo + Foi^oij + fo2^02j + X ^OiP^Oqi + X ^Os^Osj + T 0j 

q= 1 s=2 

Ppi = @ p for p > 0 


where T Q1 . is an indicator of assignment to the treatment condition for high school j, 
Y Q2 . is an indicator of cohort year for high school j, is the qth characteristic for high 
school j, ( q = 1,. . ., Q), (3 01 . is an indicator of block membership in block s for high school j 
(s = 1, . . ., 5), and r Q . is a random error term for high school j. 

Logit link function (for dichotomous dependent variables): 

’ i *' 1 o 8 (tX:) 

where y] y is the log odds of a success for student i in high school j, and u ; . is the probability 
of success for student i in high school j. 

At the student level, models include the following student baseline demographic char- 
acteristics: indicator variables for race/ethnicity, gender, eligibility for the federal school 
lunch program, Individualized Education Program status, English learner status, and grade 
level for analyses that use multiple grades. In addition, the models include measures of aca- 
demic achievement (unweighted cumulative high school grade point average and scores on 
the grade 8 state standardized math test). At the school level, models include an indicator 
of treatment condition, block membership (see figure B1 for blocks), and a prior school-lev- 
el measure of the dependent variable (for example, percentage of grade 12 students com- 
pleting the FAFSA in 2013/14 in the model predicting FAFSA completion). The variables 
included in each model are shown in table B8. 

Findings in the report are those from analyses in which missing data were imputed (with 
the specific approach to imputation depending on the particular covariate; see descrip- 
tion of variables above) and indicator variables that specified which students had missing 
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Table B8. Description of impact analysis models, by research question 

1 Research question 

Dependent variable 

Sample 

Level 1 covariates 

Level 2 covariates 1 

Confirmatory 
research question 1 

Completion of the Free 
Application for Federal 
Student Aid (binary) 

Students in grade 12 

Indicators of race/ 
ethnicity, gender, 
eligibility for the federal 
school lunch program, 
Individualized Education 

Program status, English 
learner status, state 
standardized math score, 
high school grade point 
average 

Indicator of treatment 

condition, block 
membership, percentage 
of students completing 
the FAFSA in 2013/14 

Confirmatory 
research question 2a 

ACT Engage goal striving 
scale score (continuous) 

Students in grades 10, 
11, and 12 

Indicators of race/ 
ethnicity, gender, 
eligibility for the federal 
school lunch program, 

Individualized Education 

Program status, English 
learner status, state 
standardized math score, 
high school grade point 
average, grade level 

Indicator of treatment 

condition, block 
membership, average 

ACT Engage goal striving 
scale score for grade 

10-12 students in fall 

2014/15 

Confirmatory 
research question 2b 

ACT Engage commitment 
to college scale score 
(continuous) 

Students in grades 10, 
11, and 12 

Indicators of race/ 
ethnicity, gender, 
eligibility for the federal 
school lunch program, 

Individualized Education 

Program status, English 
learner status, state 
standardized math score, 
high school grade point 
average, grade level 

Indicator of treatment 

condition, block 
membership, average 

ACT Engage commitment 
to college scale score for 
grade 10-12 students in 
fall 2014/15 

Exploratory research 
question 1 

Submission of at least 

one college application 
(binary) 

Students in grade 12 

Indicators of race/ 
ethnicity, gender, 
eligibility for the federal 
school lunch program, 
Individualized Education 

Program status, English 
learner status, state 
standardized math score, 
high school grade point 
average 

Indicator of treatment 

condition, block 
membership, percentage 
of grade 12 students 
who submitted at least 

one college application in 
2013/14 

Exploratory research 
question 2a 

Completion of the Free 
Application for Federal 
Student Aid (binary) 

ACT Engage goal striving 
scale score (continuous) 
ACT Engage commitment 
to college scale score 
(continuous) 

Submission of at least 
one college application 
(binary) 

Students who scored 

in the middle third 

on the grade 8 state 
standardized math test 

Varies by dependent 

variable 

Varies by dependent 

variable 

(continued) 
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Table B8. Description of impact analysis models, by research question (continued) 

1 Research question 

Dependent variable 

Sample 

Level 1 covariates 

Level 2 covariates 1 

Exploratory research 
question 2b 

Completion of the Free 
Application for Federal 
Student Aid (binary) 

ACT Engage goal striving 
scale score (continuous) 
ACT Engage commitment 
to college scale score 
(continuous) 

Submission of at least 

one college application 
(binary) 

Students who scored in 

the upper third of grade 8 

state standardized math 

test 

Varies by dependent 

variable 

Varies by dependent 

variable 

Exploratory research 
question 2c 

Completion of the Free 
Application for Federal 
Student Aid (binary) 

ACT Engage goal striving 
scale score (continuous) 
ACT Engage commitment 
to college scale score 
(continuous) 

Submission of at least 

one college application 
(binary) 

Students eligible for the 
federal school lunch 

program 

Varies by dependent 
variable 

Varies by dependent 
variable 

Note: See figure B1 for block membership. 

Source: Authors' study plan. 


data on particular values were included. When the same analyses were performed without 
imputed values (where cases with missing values are dropped), the magnitudes of impact 
estimates were similar to those produced with imputed data. 

Predicted probabilities for dichotomous outcomes. Supplemental analyses were conducted to 
calculate the differences in probabilities between students in Ramp-Up and comparison 
groups for the dichotomous outcomes of likelihood of completing the FAFSA and like- 
lihood of submitting at least one college application (see figure Cl in appendix C). For 
example, to calculate the change in the predicted probability of completing the FAFSA 
associated with being in the Ramp-Up group, the following steps were taken. First, the 
linear predictor of the log odds of being in the Ramp-Up group was calculated for all 
students. This linear predictor was calculated as the sum of: (1) the estimated coefficient 
of being in the Ramp-Up group multiplied by 1 minus the grand mean of being in the 
Ramp-Up group and (2) the intercept. Second, the probability of being in the Ramp-Up 
group was calculated as a transformation of the linear predictor; this probability equals 
l/[l+exp(-l*linear_predictor)]. Third, the corresponding linear predictor and probability 
of not being in the Ramp-Up group was calculated. Finally, the difference between the 
two predicted probabilities was calculated. This value indicates the difference in predict- 
ed probabilities associated with being in the Ramp-Up group for a typical student, where 
“typical” refers to a student whose values for all variables except treatment condition are at 
the grand mean among students in the model and the random student and school effects 
are equal to zero. 

Calculating Hedges’s g. Results for continuous outcomes were converted into a common 
metric (or effect size) — Fledges’s g. This statistic represents the standardized mean 
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difference, or the difference between a treatment group and a control group, gauged against 
the average standard deviation of the two groups: 

X 1 -X 2 

S= I 

I (rij - l)s 2 + (n 2 - 1 )s 2 2 
y n 1 + n 2 - 2 

where ^ is the mean for the treatment group, % 2 is the mean for the comparison group, n x 
is the student sample sizes for the treatment group, n 2 is the student sample sizes for the 
comparison group, s 2 t is the variation in outcome measure for the treatment group, and s z 2 
is the variation in outcome measure for the comparison group. 

Analyses of treatment contrast. To determine whether the supports and processes that 
comprise Ramp-Up to Readiness were different from those that comparison schools used 
to improve the college readiness of their students (or treatment contrast), the study team 
conducted three analyses. First, they examined the types of college-readiness supports 
listed on the staff survey to determine which ones were present in their school. If a staff 
member indicated that a support was present, it was coded 1; otherwise, it was coded 0. 
The study team averaged the staff responses for each school, producing a score for each 
type of support that ranged from 0 to 1 depending on staff consensus that the support was 
present. These proportions were converted into percentages and averaged across schools in 
each group (see table 1 in the main text). 

Second, the study team looked at staff responses to items on the fall and spring surveys as 
well as students’ responses to items on the student surveys related to college-focused activ- 
ities. The fall staff survey contained items on curriculum and technology (for example, 
courses that students can take for college credit, a technology platform capable of storing 
students’ postsecondary plans), college-focused professional development, and a process 
for students to create a postsecondary plan. Staff responses to items on curriculum and 
technology, professional development, and the postsecondary planning process were coded 
0 or 1, depending on whether these types of college-focused supports were present. The 
codes across staff members within schools were averaged and converted to percentages 
(representing percentage of agreement among staff that particular college-readiness sup- 
ports were present). The student surveys included items asking about college-focused staff 
interactions with students. Students’ responses also were coded so that 1 indicated that 
staff did work with students on a particular dimension of readiness or 0 if staff did not 
assist students on that dimension. Each student’s coded responses for items indicating 
“college-focused staff-student interactions” were averaged for each student and converted 
to a percentage (representing the student’s percentage agreement that staff interact with 
students around college readiness). The findings presented in figure 4 in the main text rep- 
resent the averages across students within schools and then across schools for each group. 

Third, the study team examined students’ response to survey items asking about staff inter- 
actions with students with respect to the five dimensions of college-readiness. The study 
team broke out students’ responses for each dimensions of readiness (see figure 5 in the 
main text). The averages portrayed in this figure are covariate adjusted, and the statistical 
significance for each dimension is based on hierarchical linear models that controlled for 
students within schools and school-level covariates. 
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Analysis of implementation fidelity. To answer the question of whether schools’ imple- 
mentation met the developer’s expectations, the research team systematically examined 
the components and subcomponents of Ramp-Up that the developer considers essential 
for Ramp-Up to work (figure B2): 

• Structural supports. According to the theory of action, for Ramp-Up to increase 
the likelihood of students enrolling and succeeding in college, school leaders need 
to establish the necessary structural supports for the program. These supports 
include establishing a Ramp-Up leadership team; appointing a Ramp-Up coordi- 
nator; obtaining the active participation of faculty (including having them lead 
advisories); establishing advanced courses; providing the opportunity and time for 
professional development, coordination, and preparation related to Ramp-Up; and 
implementing a technology platform for students, staff, and parents to access or 
store college-related information (such as the postsecondary plan and readiness 
rubric). 

• Professional development. According to the consortium, successful implementation 
of Ramp-Up requires that members of the Ramp-Up leadership team participate in 
eight hours of professional development led by the consortium and that Ramp-Up 
coordinators participate in an additional four hours of professional development 
led by the consortium. This professional development typically begins prior to the 
school year. Successful implementation also involves training that the leadership 
team and coordinator provide to school staff, who will serve as Ramp-Up advi- 
sors. For successful implementation, the consortium expects Ramp-Up advisors 
to receive one four-hour training session at the beginning of the school year and 
20-minute sessions during each of the nine months of the school year. 

• Curriculum delivery. The consortium requires that groups of students receive 28 
weekly lessons lasting 30 minutes each and five workshops lasting one hour each. 


Figure B2. According to the developer of Ramp-Up, for the program to have effects, 
schools need to implement the program’s key components and subcomponents 


Implementation of 
Ramp-Up to Readiness 


Structural Professional Curriculum Curriculum Postsecondary 

supports development delivery content planning tools 



Source: The University of Minnesota's College Readiness Consortium. 


B-15 










The size of the groups varies by school. To effectively lead advisories and work- 
shops, teachers need access to curriculum materials and sufficient information 
about the college-enrollment process to deliver the content. 

• Curriculum content. For Ramp-Up to improve students’ college readiness along five 
dimensions — academic readiness, admissions readiness, career readiness, financial 
readiness, and personal-social readiness (see box 1 in the main text for a descrip- 
tion of each dimension) — content material related to all five dimensions needs to 
be presented throughout the school year. 

• Postsecondary planning tools. For Ramp-Up to increase college readiness among all 
students, the program developer requires that teachers use the postsecondary plan 
and the readiness rubric to assist students in developing realistic postsecondary 
plans for achieving educational and career aspirations. 13 Fidelity is measured by 
teachers’ familiarity with the tools, the degree to which they find the tools helpful 
for their specific purpose, how often they are used, and how many students in their 
classes actually use the tools. Teachers must also share information from these 
tools with parents in two-way communication. 

Multiple indicators of each implementation component were embedded in the student and 
staff surveys. Other indicators were taken from information gathered from extant docu- 
ments provided by the consortium. 

After completing the data-coding procedures (see the previous section), an implemen- 
tation index was calculated in three steps. First, for indicators of fidelity that are based 
on staff or student surveys, the study team analyzed average responses across school-level 
respondents, and the averages were then coded 0-1, similar to the indicators based on 
qualitative data. Second, for every school, a score was calculated for each subcomponent by 
averaging across the indicators for that subcomponent (see figure B2 for subcomponents). 
Two subcomponents — presence of a Ramp-Up leadership team and time for preparation — 
were calculated using weighted averages for indicators, based on suggestions from the con- 
sortium before the study. Third, the scores were averaged up, meaning that the averages 
of subcomponents were averaged to obtain each component score, and the overall fidelity 
index score was calculated by averaging school-level component scores. The index, there- 
fore, represents the proportion of indicators of Ramp-Up that schools have successfully 
implemented (these indices are presented as percentages by multiplying by 100). 

Finally, based on these fidelity index scores and cutpoints established by the consortium, 
schools’ overall implementation, implementation of components, and implementation of 
subcomponents were classified into three categories. 14 Schools having fidelity index scores 
less than 60 percent were judged as showing inadequate implementation, those having 
scores in the range of 60-89 percent were judged as showing adequate implementation, 
and those having index scores of 90 percent or higher were judged as showing excellent 
implementation. Thus, schools are given labels of inadequate, adequate, or excellent for 
each subcomponent and component and for the entire fidelity of implementation model. 


B-16 


Appendix C. Detailed results 


The tables in this appendix present complete results from analyses that address the confir- 
matory, exploratory, and implementation research questions. 


Table Cl. At baseline, Ramp-Up schools and comparison schools had student populations with similar 
demographic characteristics and student achievement, 2013/14 


Ramp-Up schools 

Comparison schools 


Characteristic 

Number 

Mean 

Standard 

deviation 

Number 

Mean 

Standard 

deviation 

Standardized 

difference 8 

School enrollment 

25 

474 

439 

25 

505.9 

432 

-0.07 

Students eligible for the federal school lunch program (%) 

25 

34.9 

19.6 

24 

33.4 

16.9 

0.08 

English learner students (%) 

25 

1.1 

2.6 

24 

4.5 

17.8 

-0.33 

Students with individualized educational programs (%) 

25 

14.1 

5.2 

24 

13.3 

5.5 

0.15 

American Indian/Alaska Native students (%) 

25 

10.1 

22.0 

24 

3.9 

8.0 

0.41 

Asian or Pacific Islander students (%) 

25 

2.4 

4.0 

24 

2.0 

3.5 

0.11 

Hispanic students (%) 

25 

4.5 

5.7 

24 

3.6 

3.8 

0.19 

Black students (%) 

25 

3.6 

8.9 

24 

5.4 

17.3 

-0.14 

White students (%) 

25 

79.3 

24.4 

24 

85.2 

20.0 

-0.27 

Students with state standardized math test score from 
grade ll b (%) 

22 

1,146 

6.5 

23 

1,146 

6.4 

0.05 

Students with state standardized reading test score 
from grade 10 b (%) 

23 

1,049 

5.3 

22 

1,050 

6.3 

-0.12 

Four-year graduation rate (%) 

25 

82.7 

21.2 

24 

88.8 

11.8 

-0.37 

ACT composite score 

19 

21.6 

1.3 

23 

21.7 

2.1 

0.00 


a. Standardized using Cohen's d. None of the differences between Ramp-Up schools and comparison schools was statistically 
significant. 

b. State standardized test refers to the Minnesota Comprehensive Assessment Series II and the Wisconsin Concepts and Knowledge 
Examination. 

Source: Authors’ analyses of data from Minnesota Department of Education (n.d.) and Wisconsin Department of Public Instruction (n.d.). 
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Table C2. Regression models predicting self-reported completion of the Free Application for Federal Student Aid, ACT Engage goal 
striving and commitment to college scale scores, and submission of at least one college application among all eligible students, by 
student and school characteristics 


Characteristic 

Self reported FAFSA 
completion 3 

ACT Engage goal 
striving scale score b 

ACT Engage 
commitment to 
college scale score b 

Submitted at least one 
college application 3 

Odds 

ratio 

95 percent 
confidence 
interval 

Beta 

Standard 

error 

Beta 

Standard 

error 

Odds 

ratio 

95 percent 
confidence 
interval 

Student characteristics 

Grade level 

na 

na 

0.04 

0.18 

-0.20 

0.19 

na 

na 

Eligible for the federal school lunch program 

0.87 

0.68, 1.12 

-0.05 

0.36 

-0.39 

0.41 

0.79 

0.52, 1.19 

Indicator of missing data on eligibility for the federal school lunch program 

1.53*** 

1.24, 1.88 

0.95 

0.54 

1.40*** 

0.35 

1.01 

0.41, 2.51 

Female 

0.98 

0.81, 1.19 

0.04 

0.27 

0.73* 

0.32 

1.33 

0.95, 1.86 

Indicator of missing data on gender 

2 08*** 

1.49, 2.91 

0.15 

0.84 

0.45 

0.52 

4.93 

0.73, 33.28 

English learner student indicator 

1.07 

0.35, 3.24 

-0.10 

0.80 

-0.10 

1.06 

1.59 

0.52, 4.85 

Individual education program indicator 

0.49*** 

0.35, 0.67 

0.18 

0.56 

-0.43 

0.71 

0.83 

0.50, 1.36 

American Indian/Alaska Native 

1.03 

0.52, 2.03 

0.29 

1.15 

0.39 

1.18 

1.35 

0.50, 3.64 

Black 

1.28 

0.87, 1.90 

0.71 

0.70 

1.75 

0.93 

0.91 

0.35, 2.41 

Hispanic 

1.14 

0.68, 1.92 

0.49 

0.86 

0.36 

0.89 

1.22 

0.60, 2.47 

Indicator of missing data on race/ethnicity 

0.49* 

0.28, 0.85 

2.72*** 

0.53 

1.04 

0.59 

0.13* 

0.02, 0.89 

Grade 8 MCA-ll d math score 

1.04 

0.93, 1.17 

0.13 

0.16 

0.08 

0.19 

0.97 

0.78, 1.22 

Indicator of missing MCA-ll d math score 

1.21 

0.95, 1.55 

0.07 

0.38 

0.38 

0.48 

0.67 

0.43, 1.05 

Cumulative grade point average, fall 2014 e 

3.05*** 

2.58, 3.61 

1.01*** 

0.28 

1.71*** 

0.33 

3.88*** 

3.01, 5.01 

Indicator of missing fall 2014 grade point average 

1.18 

0.30, 4.59 

-2.92** 

0.90 

-1.87* 

0.88 

2.76 

0.21, 35.86 

ACT Engage goal striving scale score, fall 2014 

na 

na 

0.70*** 

0.03 

na 

na 

na 

na 

ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.71*** 

0.03 

na 

na 

School characteristics 

Treatment condition 

1.23 

0.83, 1.82 

0.11 

0.43 

0.09 

0.60 

1.26 

0.66, 2.41 

Block 

1.00 

0.85, 1.16 

-0.10 

0.20 

-0.12 

0.24 

1.17 

0.91, 1.49 

Percentage of grade 12 students completing the FAFSA in 2013/14 

0.75 

0.09, 6.05 

na 

na 

na 

na 

na 

na 

Average ACT Engage goal striving scale score, fall 2014 

na 

na 

-0.06 

0.22 

na 

na 

na 

na 

Average ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.02 

0.19 

na 

na 

Percentage of grade 12 students who submitted at least one college 









application in 2013/14 

na 

na 

0.04 

0.18 

-0.20 

0.19 

2.61 

0.39, 17.37 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001 

FAFSA is the Free Application for Federal Student Aid. na is not applicable because the covariate was not used in the corresponding regression model. MCA is the Minnesota Com- 
prehensive Assessment. 

Note: Numbers in table are test coefficients from hierarchical linear models, with students nested in schools. 

a. Based on 2,128 grade 12 students in 43 schools. 

b. Based on 1,776 students in 45 schools. 

c. Based on 1,526 grade 12 students in 36 schools. 

d. Refers to the Wisconsin Concepts and Knowledge Examination for students in Wisconsin schools. 

e. Data indicate the change based on a one-unit increase in grade point average (for example, from a 3.0 to a 4.0). So, the odds of completing the FAFSA increase by 3.05, meaning 
that students with a 4.0 grade point average are a little more than three times as likely as students with a 3.0 grade point average to self-report completing the FAFSA. 

Source: Authors' calculations based on data from Ramp-Up and comparison schools and the Federal Student Aid website (https://studentaid.ed.gov/sa/about/data-center/student/ 
application-volume/fafsa-completion-high-school). 
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Table C3. Regression models predicting self-reported completion of the Free Application for Federal Student Aid, ACT Engage goal 
striving and commitment to college scale scores, and submission of at least one college application among students between the 33rd 
and 66th percentiles of math achievement 


Characteristic 

Self reported FAFSA 
completion 3 

ACT Engage goal 
striving scale score b 

ACT Engage 
commitment to 
college scale score b 

Submitted at least one 
college application^ 

Odds 

ratio 

95 percent 
confidence 
interval 

Beta 

Standard 

error 

Beta 

Standard 

error 

Odds 

ratio 

95 percent 
confidence 
interval 

Student characteristics 

Grade level 

na 

na 

-0.17 

0.27 

-0.69** 

0.22 

na 

na 

Eligible for the federal school lunch program 

0.69 

0.40, 1.17 

-0.31 

0.38 

0.15 

0.56 

0.43 

0.21, 0.87 

Indicator of missing data on eligibility for the federal school lunch program 

1.95** 

1.24, 3.07 

0.21 

1.32 

1.71 

1.22 

2.45* 

0.47, 12.90 

Female 

1.09 

0.78, 1.51 

-0.11 

0.36 

0.54 

0.42 

na 

na 

Indicator of missing data on gender 

1.69 

0.86, 3.33 

0.74 

0.55 

1.99** 

0.69 

na 

na 

English learner student indicator 

na 

na 

-0.63 

1.05 

3.16 

2.52 

na 

na 

Individual education program indicator 

0.52 

0.13, 1.99 

0.45 

0.98 

-1.61 

1.51 

0.50 

0.12, 2.09 

American Indian/Alaska Native 

1.03 

0.29, 3.62 

-1.38 

0.83 

1.26 

1.23 

na 

na 

Black 

3.74 

0.36, 39.21 

-0.85 

0.95 

0.18 

1.68 

na 

na 

Hispanic 

0.61 

0.35, 1.06 

2.00 

2.00 

-0.11 

2.06 

na 

na 

Indicator of missing data on race/ethnicity 

na 

na 

2.13** 

0.79 

1.76 

0.95 

na 

na 

Grade 8 MCA-ll d math score 

na 

na 

na 

na 

na 

na 

1.18 

0.27, 5.12 

Indicator of missing MCA-ll d math score 

na 

na 

na 

na 

na 

na 

na 

na 

Cumulative grade point average, fall 2014 

na 

na 


0.40 

2 69*** 

0.47 

na 

na 

Indicator of missing fall 2014 grade point average 

na 

na 

0.15 

2.37 

0.15 

1.94 

na 

na 

ACT Engage goal striving scale score, fall 2014 

na 

na 

0.74*** 

0.03 

na 

na 

na 

na 

ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.72*** 

0.04 

na 

na 

School characteristics 

Treatment condition 

0.72 

0.48, 1.08 

0.41 

0.45 

-0.63 

0.58 

1.66 

0.61, 4.52 

Block 

1.09 

0.88, 1.34 

-0.48* 

0.22 

-0.46 

0.26 

1.35 

0.90, 2.02 

Percentage of grade 12 students completing the FAFSA in 2013/14 

14.23* 

1.13, 179.22 

na 

na 

na 

na 

na 

na 

Average ACT Engage goal striving scale score, fall 2014 

na 

na 

-0.15 

0.19 

na 

na 

na 

na 

Average ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.17 

0.19 

na 

na 

Percentage of grade 12 students who submitted at least one college 









application in 2013/14 

na 

na 

na 

na 

na 

na 

3.36 

0.19, 61.01 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001 

FAFSA is the Free Application for Federal Student Aid. na is not applicable because the covariate was not used in the corresponding regression model. MCA is the Minnesota Com- 
prehensive Assessment. 

a. Based on 534 grade 12 students in 41 schools. 

b. Based on 578 students in 38 schools. 

c. Based on 374 grade 12 students in 27 schools. 

d. Refers to the Wisconsin Concepts and Knowledge Examination for students in Wisconsin schools. 

Source: Authors’ calculations based on data from Ramp-Up and comparison schools and the Federal Student Aid website (https://studentaid.ed.gov/sa/about/data-center/student/ 
application-volume/fafsa-completion-high-school). 
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Table C4. Regression models predicting self-reported completion of the Free Application for Federal Student Aid, ACT Engage goal 
striving and commitment to college scale scores, and submission of at least one college application among students at or above the 
66th percentile of math achievement 


Characteristic 

Self-reported FAFSA 
completion 3 

ACT Engage goal 
striving scale score b 

ACT Engage 
commitment to 
college scale score b 

Submitted at least one 
college application^ 

Odds 

ratio 

95 percent 
confidence 
interval 

Beta 

Standard 

error 

Beta 

Standard 

error 

Odds 

ratio 

95 percent 
confidence 
interval 

Student characteristics 

Grade level 

na 

na 

-0.12 

0.36 

0.02 

0.34 

na 

na 

Eligible for the federal school lunch program 

0.49* 

0.27, 0.90 

-0.21 

0.83 

-0.10 

0.78 

0.57 

0.17, 1.89 

Indicator of missing data on eligibility for the federal school lunch program 

0.76 

0.34, 1.72 

0.24 

1.54 

-0.33 

1.28 

0.60 

0.09, 3.98 

Female 

1.53* 

1.02, 2.29 

1.28* 

0.59 

0.21 

0.56 

2.13 

0.65, 6.95 

Indicator of missing data on gender 

3.08 

0.87, 10.83 

-0.62 

1.91 

0.59 

1.51 

0.72 

0.06, 5.53 

English learner student indicator 

na 

na 

na 

na 

na 

na 

na 

na 

Individual education program indicator 

na 

na 

-1.81 

3.22 

-2.69 

3.02 

0.58 

0.06, 5.53 

American Indian/Alaska Native 

na 

na 

na 

na 

na 

na 

na 

na 

Black 

na 

na 

na 

na 

na 

na 

na 

na 

Flispanic 

na 

na 

na 

na 

na 

na 

na 

na 

Indicator of missing data on race/ethnicity 

na 

na 

na 

na 

na 

na 

na 

na 

Grade 8 MCA-ll d math score 

na 

na 

na 

na 

na 

na 

1.10 

0.39, 3.12 

Indicator of missing MCA-ll d math score 

na 

na 

na 

na 

na 

na 

na 

na 

Cumulative grade point average, fall 2014 

na 

na 

na 

na 

na 

na 

4.58*** 

2.07, 10.13 

Indicator of missing fall 2014 grade point average 

na 

na 

na 

na 

na 

na 

na 

na 

ACT Engage goal striving scale score, fall 2014 

na 

na 

0.78*** 

0.04 

na 

na 

na 

na 

ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.89*** 

0.04 

na 

na 

School characteristics 

Treatment condition 

1.38 

0.85, 2.25 

0.16 

0.81 

-0.28 

0.66 

1.19 

0.29, 5.00 

Block 

0.81 

0.65, 1.01 

-0.43 

0.35 

-0.35 

0.29 

0.80 

0.45, 1.42 

Percentage of grade 12 students completing the FAFSA in 2013/14 

0.06 

0.00, 1.45 

na 

na 

na 

na 

na 

na 

Average ACT Engage goal striving scale score, fall 2014 

na 

na 

0.40 

0.36 

na 

na 

na 

na 

Average ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.12 

0.20 

na 

na 

Percentage of grade 12 students who submitted at least one college 









application in 2013/14 

na 

na 

na 

na 

na 

na 

3.19 

0.07, 152.27 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001 

FAFSA is the Free Application for Federal Student Aid. na is not applicable because the covariate was not used in the corresponding regression model. MCA is the Minnesota Com- 
prehensive Assessment. 

a. Based on 445 grade 12 students in 39 schools. 

b. Based on 390 students in 44 schools. 

c. Based on 297 grade 12 students in 26 schools. 

d. Refers to the Wisconsin Concepts and Knowledge Examination for students in Wisconsin schools. 

Source: Authors' calculations based on data from Ramp-Up and comparison schools and the Federal Student Aid website (https://studentaid.ed.gov/sa/about/data-center/student/ 
application-volume/fafsa-completion-high-school). 
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Table C5. Regression models predicting self-reported completion of the Free Application for Federal Student Aid, ACT Engage goal 
striving and commitment to college scores, and submission of at least one college application among students eligible for the federal 
school lunch program 


Characteristic 

Self-reported FAFSA 
completion 3 

ACT Engage goal 
striving scale score b 

ACT Engage 
commitment to 
college scale score b 

Submitted at least one 
college application^ 

Odds 

ratio 

95 percent 
confidence 
interval 

Beta 

Standard 

error 

Beta 

Standard 

error 

Odds 

ratio 

95 percent 
confidence 
interval 

Student characteristics 

Grade level 

na 

na 

-0.47 

0.33 

-0.29 

0.36 

na 

na 

Eligible for the federal school lunch program 

na 

na 

na 

na 

na 

na 

na 

na 

Indicator of missing data on eligibility for the federal school lunch program 

na 

na 

na 

na 

na 

na 

na 

na 

Female 

0.80 

0.52, 1.22 

0.08 

0.56 

1.31* 

0.62 

2.32** 

1.29, 4.15 

Indicator of missing data on gender 

1.10 

0.16, 7.43 

1.01 

1.75 

-0.49 

1.80 

0.71 

0.07, 7.01 

English learner student indicator 

0.80 

0.29, 2.20 

0.29 

1.29 

-0.07 

1.36 

1.43 

0.36, 5.72 

Individual education program indicator 

0.39* 

0.19, 0.81 

-0.24 

0.75 

-1.10 

0.82 

0.60 

0.28, 1.30 

American Indian/Alaska Native 

0.38 

0.14, 1.08 

-0.05 

1.08 

-0.35 

1.14 

0.66 

0.20, 2.20 

Black 

1.19 

0.54, 2.65 

1.18 

1.13 

3.33** 

1.22 

0.59 

0.17, 2.12 

Hispanic 

1.10 

0.60, 1.99 

0.12 

0.94 

0.72 

1.02 

0.55 

0.24, 1.29 

Indicator of missing data on race/ethnicity 

0.00 

0.00, 1.00 

1.36 

1.88 

-0.11 

1.99 

0.25 

0.01, 5.16 

Grade 8 MCA-ll d math score 

0.83 

0.61, 1.13 

0.64 

0.36 

0.52 

0.39 

1.13 

0.26, 1.22 

Indicator of missing MCA-ll d math score 

1.02 

0.55, 1.90 

-0.24 

0.74 

-1.11 

0.80 

0.57 

0.26, 1.22 

Cumulative grade point average, fall 2014 

3.16*** 

2.21, 4.52 

0.57 

0.41 

1.13* 

0.46 

na 

na 

Indicator of missing fall 2014 grade point average 

0.92 

0.11, 7.69 

-3.68* 

1.73 

-1.03 

1.82 

na 

na 

ACT Engage goal striving scale score, fall 2014 

na 

na 

0.69*** 

0.03 

na 

na 

na 

na 

ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

0.72*** 

0.03 

na 

na 

School characteristics 

Treatment condition 

1.33 

0.68, 2.60 

-0.08 

0.69 

0.28 

0.71 

i.ii 

0.44, 2.78 

Block 

1.15 

0.87, 1.53 

-0.11 

0.26 

-0.11 

0.27 

1.51* 

1.07, 2.14 

Percentage of grade 12 students completing the FAFSA in 2013/14 

0.26 

0.02, 3.21 

na 

na 

na 

na 

na 

na 

Average ACT Engage goal striving scale score, fall 2014 

na 

na 

-0.11 

0.23 

na 

na 

na 

na 

Average ACT Engage commitment to college scale score, fall 2014 

na 

na 

na 

na 

-0.08 

0.18 

na 

na 

Percentage of grade 12 students who submitted at least one college 









application in 2013/14 

na 

na 

na 

na 

na 

na 

7.07 

0.55, 90.43 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001 

FAFSA is the Free Application for Federal Student Aid. na is not applicable because the covariate was not used in the corresponding regression model. MCA is the Minnesota Com- 
prehensive Assessment. 

a. Based on 515 grade 12 students in 43 schools. 

b. Based on 636 students in 44 schools. 

c. Based on 365 grade 12 students in 33 schools. 

d. Refers to the Wisconsin Concepts and Knowledge Examination for students in Wisconsin schools. 

Source: Authors’ calculations based on data from Ramp-Up and comparison schools and the Federal Student Aid website (https://studentaid.ed.gov/sa/about/data-center/student/ 
application-volume/fafsa-completion-high-school). 





Table C6. Ramp-Up schools’ implementation scores show adequate fidelity overall, but some 
components and subcomponents were inadequately implemented, 2014/15 


Fidelity classifications 

Average implementation scores (percent of schools) 


Component/subcomponent 

All 

schools 

(average) 

Fidelity 

score 3 

Lowest 

score 

Highest 

score 

Excellent 

Adequate 

Inadequate 

Overall fidelity 

71 

A 

57 

78 

0 

96 

4 

Structural supports 

77 

A 

56 

90 

4 

92 

4 

Ramp-Up leadership team 

73 

A 

50 

100 

24 

52 

24 

Ramp-Up coordinator 

88 

A 

39 

100 

64 

32 

4 

Ramp-Up advisors 

94 

E 

77 

100 

96 

4 

0 

Advanced courses 

86 

A 

50 

100 

52 

40 

8 

Support for professional development 

71 

A 

0 

100 

36 

32 

32 

Time to implement 

50 

1 

8 

90 

4 

12 

84 

Technology platform b 

22 

1 

6 

86 

0 

4 

96 

Professional development 11 

87 

A 

73 

100 

32 

64 

0 

For leadership team/coordinator 

93 

E 

70 

100 

72 

28 

0 

For teachers/advisors b 

79 

A 

60 

97 

21 

79 

0 

Curriculum delivery 

75 

A 

52 

86 

0 

96 

4 

Materials 

74 

A 

41 

91 

8 

80 

12 

Information on advisories and workshops 

76 

A 

43 

87 

0 

88 

12 

Advisories 

92 

E 

67 

100 

64 

36 

0 

Workshops 0 

54 

1 

13 

88 

0 

50 

50 

Curriculum content 11 

62 

A 

49 

72 

0 

63 

37 

Academic readiness 11 

52 

1 

38 

70 

0 

17 

83 

Admissions readiness 0 

62 

A 

53 

73 

0 

67 

33 

Career readiness 11 

60 

A 

39 

81 

0 

42 

58 

Financial readiness 11 

50 

1 

33 

82 

0 

12 

88 

Personal-social readiness 0 

86 

A 

69 

94 

20 

80 

0 

Postsecondary planning tools 

53 

1 

39 

71 

0 

12 

88 

Use of postsecondary plan 

62 

A 

48 

81 

0 

60 

40 

Use of readiness rubric 

64 

A 

41 

88 

0 

52 

48 

Parent communication 

12 

1 

0 

44 

0 

0 

25 


Note: The indicators for this component come exclusively from the student survey, which was administered to 24 of the 25 Ramp-Up 
schools because one Ramp-Up school was unable to administer that survey prior to the end of the school year. 

a. E indicates excellent implementation (scores greater than or equal to 90 percent); A indicates adequate implementation (fidelity 
scores between 60 percent and 89 percent); / indicates inadequate implementation (fidelity scores less than 60 percent). 

b. Based on 24 schools. 

c. Based on 22 schools. 

Source: Authors' calculations based on student and staff survey data from fall 2014 and spring 2015. 
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Figure Cl. Change in predicted probability of completing the Free Application 
for Federal Student Aid and submitting at least one college application between 
students in Ramp-Up schools and students in comparison schools, 2014/15 
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Note: Positive percentages indicate that students in Ramp-Up schools were more likely than students in 
comparison schools to perform the enrollment-related action; negative percentages indicate that students in 
Ramp-Up schools were less likely. None of the differences between Ramp-Up schools and comparison schools 
was statistically significant. 

Source: Authors' calculations based on extant student data and student survey data from fall 2014 and spring 
2015. 
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Notes 


1. The What Works Clearinghouse is an initiative of the U.S. Department of Educa- 
tion Institute of Education Sciences to develop standards by which to gauge whether 
studies are rigorous enough to detect causal relationships and to evaluate the strength 
of evidence on the effectiveness of programs. More information is available at http:// 
ies.ed.gov/ncee/wwc/. 

2. The alliance consists of representatives of state affiliates of the National College 
Access Network, community college boards of directors, state education agencies, and 
state higher education agencies. Members come from six of the seven Regional Educa- 
tional Laboratory Midwest Region states (Illinois, Indiana, Iowa, Michigan, Minneso- 
ta, and Ohio). 

3. The original impact study sample included 50 high schools, 25 randomly assigned to 
each group. One school in the comparison group dropped out of the study, leaving 49 
schools in the final sample. 

4. The number of schools (18 Ramp-Up schools and 19 comparison schools) included in 
this analysis is smaller than the total number of participating schools because not all 
schools administered the student survey. 

5. The comparisons between the two groups of schools for these subsamples may have 
been underpowered. That is, there may have been too few schools and students in 
these samples to detect smaller effects (see table B7 in appendix B for the minimum 
detectable effect sizes for these subsamples). 

6. Exact school details are omitted to protect schools’ identities. 

7. The study team substantiated this assumption by comparing data in the study sample 
with school-level enrollment data on Individualized Education Programs and English 
learner status maintained by the state. 

8. For schools with fewer than 30 students in a grade, the study team requested that all 
students in the grade complete the student survey. 

9. The ACT Engage survey was administered to only a sample of grade 12 students 
because of resource constraints. 

10. This number is lower than expected because not all schools had 30 students in each 
grade. In those cases, all students were selected to complete the ACT Engage survey. 

11. In the domain of social engagement the ACT Engage scales are social connection and 
social activity. In the domain of self-regulation the scales are academic self-confidence 
and steadiness. In the domain of motivation and skills the scales are academic disci- 
pline, general determination, goal striving, commitment to college, study skills, and 
communication skills. Students took the complete assessment (all 10 scales) because 
that is the form in which ACT offers it. 

12. These studies examine the predictive validity of an earlier version of ACT Engage 
known as the Student Readiness Inventory among two- and four-year college students. 
No published data are available for high school samples. 

13. The consortium recommends that Ramp-Up schools store and maintain students’ 
postsecondary plans using an electronic platform. Paper versions of the postsecondary 
plan are two pages, with fields in which students enter information related to each 
dimension of readiness. Students record their career aspirations and the colleges that 
match their interests, and then they offer a course of study geared to their career 
choice. Students also enter courses that will help them meet admission criteria sought 
by their college of choice. The readiness rubric is a self-assessment on which students 
rate their progress on following through on their postsecondary plan. Paper copies of 
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both the postsecondary plan and readiness rubric are available from the authors or 
from the College Readiness Consortium. 

14. The consortium established these cutpoints for schools during their first year of imple- 
mentation. Hypothetically, the cutpoints would be higher for schools that have been 
implementing Ramp-Up longer. 
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Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 



Stated Briefly 

Summaries of research findings for specific audiences 


Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 




